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Executive  Summary 


The  National  Defense  Industrial  Association  Systems  Engineering  Division  (NDIA-SED)  collabo¬ 
rated  with  the  Institute  of  Electrical  and  Electronic  Engineers  Aerospace  and  Electronic  Systems 
Society  (lEEE-AESS)  and  the  Software  Engineering  Institute  (SEI)  of  Carnegie  Mellon®  to  obtain 
quantitative  evidence  of  the  benefit  of  systems  engineering  (SE)  best  practices  on  project  perfor¬ 
mance.  The  team  developed  and  executed  this  survey  of  system  developers  to  identify  SE  best 
practices  used  on  projects,  collect  performance  data  on  these  projects,  and  identify  relationships 
between  the  application  of  these  SE  best  practices  and  project  performance. 


The  study  found  clear  and  significant  relationships  between  the  application  of  SE  best  practices  to 
projects  and  the  performance  of  those  projects,  as  seen  in  the  mosaic  chart  in  Figure  1  and  as  ex¬ 
plained  below. 


Gamma  =  0.49  p-value  <  0.001 


Higher 

Perf 


Middle 

Perf 


Lower 

Perf 


All  Projects 


Figure  1:  Project  Performance  vs.  Total  SE  Capability 

The  left  column  represents  projects  deploying  lower  levels  of  SE,  as  measured  by  assessing  the 
quantity  and  quality  of  specific  SE  work  products.  Among  these  projects,  only  15%  delivered 
higher  levels  of  project  performance,  as  measured  by  satisfaction  of  budget,  schedule,  and  tech¬ 
nical  requirements.  Within  this  group,  52%  delivered  lower  levels  of  project  performance. 

The  second  column  represents  those  projects  deploying  moderate  levels  of  SE.  Among  these  pro¬ 
jects,  24%  delivered  higher  levels  of  project  performance  and  29%  delivered  lower  levels  of  per¬ 
formance. 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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The  third  column  represents  projects  deploying  higher  levels  of  SE.  Among  these  projects,  the 
number  delivering  higher  levels  of  project  performance  increased  to  57%,  while  those  delivering 
lower  levels  decreased  to  20%. 

In  addition  to  the  mosaic  chart’s  graphical  representation  of  the  relationship,  we  also  employed 
statistical  measures  to  characterize  it.  Goodman  and  KruskaTs  Gamma  expresses  the  strength  of 
the  relationship  between  two  ordinal  variables.  Gamma  values  near  (-1)  indicate  a  very  strong 
opposing  relationship;  values  near  0  indicate  a  weak  or  non-existent  relationship;  values  near  (+1) 
indicate  a  very  strong  supporting  relationship.  The  gamma  value  of  0.49  in  Figure  1  represents  a 
very  strong  relationship  between  SE  deployment  and  project  performance. 

To  further  understand  the  relationship  between  SE  capability  and  project  performance,  the  ques¬ 
tionnaire’s  assessment  of  SE  capability  addressed  the  project’s  use  of  SE  best  practices  in  1 1 
management  and  technical  process  groups.  Details  regarding  the  contents  of  these  process  groups 
are  described  in  this  report.  Responses  were  analyzed  to  identify  relationships  between  project 
performance  and  the  project’s  use  of  SE  best  practices  in  each  of  the  process  groups.  Table  land 
Figure  2  summarize  these  relationships. 


Table  1:  Summary  of  Project  Performance  versus  Systems  Engineering  Capabilities 


Driver' 

Gamma 

Section 

SEC-Total:  total  deployed  SE 

+0.49  =>  Very  strong  positive 

5.4.1 

SEC-PP:  project  planning 

+0.46  =>  Very  strong  positive 

5.4.3 

SEC-REQ:  requirements  development  and  management 

+0.44  =>  Very  strong  positive 

5.4.2 

SEC-VER:  verification 

+0.43  =>  Very  strong  positive 

5.4.7 

SEC-ARCH:  product  architecture 

+0.41  =>  Very  strong  positive 

5.4.4 

SEC-CM:  configuration  management 

+0.38  =>  Strong  positive 

5.4.11 

SEC-TRD:  trade  studies 

+0.38  =>  Strong  positive 

5.4.5 

SEC-PMC:  project  monitoring  and  control 

+0.38  =>  Strong  positive 

5.4.9 

SEC-PI:  product  integration 

+0.33  =>  Strong  positive 

5.4.6 

SEC-VAL:  validation 

+0.33  =>  Strong  positive 

5.4.8 

SEC-RSKM:  risk  management 

+0.21  =>  Moderate  positive 

5.4.10 

SEC-IPT:  integrated  product  team  utilization 

+0. 1 8  =>  Weak  positive 

5.4.3 

Avoid  overinterpreting  the  meaning  of  the  Driver  categories.  For  example,  the  Project  Planning  category  in¬ 
cludes  elements  of  project  planning,  but  is  not  a  comprehensive  compilation  of  all  project  planning  activities. 
The  project  challenge  category  (as  shown  in  Figure  2  and  Table  2)  Includes  a  number  of  factors  that  influence 
the  difficulty  of  a  project,  but  is  not  a  comprehensive  assessment  of  all  factors  contributing  to  a  project's  chal¬ 
lenge.  To  better  understand  the  listed  relationships,  please  refer  to  the  report  sections  listed  in  the  last  column 
that  describe  the  contents  of  each  category. 
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Performance  vs.  SE  Capability  -  All  Projects 

-0.3  -0.2  -0.1  0.0  0.1  0.2  0.3  0.4  0.5  0.6  0.7 
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Project  Planning 
Req'ts  Dev't  &  Mg't 
Verification 
Product  Architecture 
Configuration  Mg't 
Trade  Studies 
Monitor  &  Control 
Validation 
Product  Integration 
Risk  Management 
Integ.  Product  Teams 
Project  Challenge 
Prior  Experience 


Figure  2:  Project  Performance  vs.  SE  Capabilities  and  Drivers 

The  survey  also  examined  the  relationships  between  project  performance  and  other  factors  such  as 
project  challenge  and  prior  experience.  Table  2  summarizes  the  relationships  for  these  factors. 

Table  2:  Summary  of  Project  Performance  versus  Other  Factors 


Driver^ 

Gamma 

Section 

PC:  Project  challenge 

-0.26  =>  Moderate  negative 

5.3 

EXP:  Prior  experience 

+0.36  =>  Strong  positive 

5.5.1 

The  importance  of  implementing  systems  engineering  best  practices  becomes  even  more  evident 
when  we  consider  differences  in  project  challenge  (PC).  Such  practices  are  particularly  important 
for  projects  that  face  more  difficult  challenges  in  implementing  their  deliverables  (Figure  3). 

The  chart  on  the  left  side  of  Figure  3  shows  the  relationship  between  SEC-Total  and  Perf  for  pro¬ 
jects  with  lower  PC.  ft  shows  a  strong  supporting  relationship  between  SEC-Total  and  Perf,  with 
the  percentage  of  projects  delivering  higher  performance  changing  from  23  %  to  23  %  to  52  %  as 
SEC-Total  increased  from  lower  to  middle  to  higher. 

Similarly,  the  percentage  of  projects  delivering  lower  performance  decreased  from  32%  to  19%  to 
12%  as  SEC-Total  increased.  Thus,  for  the  lower  challenge  projects,  the  likelihood  of  delivering 
higher  performance  more  than  doubled  with  improved  SEC-Total,  while  those  delivering  lower 
performance  was  reduced  to  less  than  half  This  relationship  is  characterized  by  a  Gamma  value 
of  +0.34  and  a  low  p-value  of  0.029. 
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Low  PC 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=22)  (n=26)  (n=25) 


Gamma  =  0.34  p-value  =  0.029 


High  PC 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=26)  (n=23)  (n=26) 


Gamma  =  0.62  p-value  <  0.001 


Figure  3:  Project  Performance  vs.  Total  SE  Capability  controlled  by  Project  Challenge 

The  chart  on  the  right  side  of  Figure  3  shows  a  very  strong  relationship  hQtmQQn  SEC-Total  and 
Perf  for  those  projects  with  higher  PC.  The  percentage  of  projects  delivering  higher  performance 
increased  from  8%  to  26%  to  62%  as  SEC-Total  increased  from  lower  to  middle  to  higher.  Addi¬ 
tionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  69%  to  39%  to 
27%  as  SEC-Total  increased. 

Thus,  for  these  higher  challenge  projects,  the  likelihood  of  delivering  higher  performance  in¬ 
creased  more  than  sevenfold  and  that  of  delivering  lower  performance  decreased  by  almost  two- 
thirds  with  improved  SEC-Total.  This  relationship  is  characterized  by  a  Gamma  value  of  +0.62 
and  a  very  low  p-value  less  than  0.001 . 

The  meaning  of  this  information  is  clear: 

Projects  that  properly  apply  systems  engineering  best  practices  perform  better  than  projects 
that  do  not. 

This  report  identifies  the  SE  process  groups  that  have  the  strongest  relationships  to  project  per¬ 
formance.  It  also  shows  that  more  challenging  projects  tend  to  perform  worse  than  less  challeng¬ 
ing  projects.  However  projects  that  face  less  challenge  still  tend  to  benefit  from  implementing 
systems  engineering  best  practices.  Moreover  the  impact  of  employing  systems  engineering  best 
practices  is  even  greater  for  more  challenging  projects. 

With  this  knowledge,  system  acquirers  and  system  developers  can  inform  their  judgments  regard¬ 
ing  the  application  of  SE  to  their  projects  and  improve  their  SE  practices  to  further  enhance  pro¬ 
ject  outcomes. 
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Abstract 


This  report  summarizes  the  resuits  of  a  survey  that  had  the  goai  of  quantifying  the  connection  be¬ 
tween  the  appiication  of  systems  engineering  (SE)  best  practices  to  projects  and  programs  and  the 
performance  of  those  projects  and  programs.  The  survey  popuiation  consisted  of  projects  and  pro¬ 
grams  executed  by  system  deveiopers  reached  through  the  Nationai  Defense  Industriai  Associa¬ 
tion  Systems  Engineering  Division  (NDIA-SED),  the  Institute  of  Eiectricai  and  Eiectronics  Engi¬ 
neers  Aerospace  and  Eiectronic  Systems  Society  (lEEE-AESS),  and  the  Intemationai  Councii  on 
Systems  Engineering  (INCOSE).  Anaiysis  of  survey  responses  reveaied  strong  statisticai  reiation- 
ships  between  project  performance  and  severai  categories  of  specific  SE  best  practices.  The  sur¬ 
vey  resuits  show  notabie  differences  in  the  reiationship  between  SE  best  practices  and  perfor¬ 
mance  between  more  chaiienging  and  iess  chaiienging  projects.  The  statisticai  reiationship  with 
project  performance  is  quite  strong  for  survey  data  of  this  kind  when  both  SE  capabiiity  and  pro¬ 
ject  chaiienge  are  considered  together. 
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1  Introduction 


1.1  Background 

An  understanding  of  the  value  of  systems  engineering  (SE)  is  necessary  to  justify  a  project’s  in¬ 
vestment  in  SE  resources  and  activities.  And  yet,  quantitative  data  showing  the  contributions  of 
SE  to  project  success  are  sparse.  Prior  research  by  Gruhl  showed  that  NASA  projects  that  spent  a 
larger  percentage  of  their  efforts  in  the  early  phases  of  the  project  exhibited  smaller  cost  over¬ 
runs.^  For  NASA  programs,  SE  funding  is  higher  as  a  percentage  of  the  total  funds  in  early  phas¬ 
es,  than  in  later  phases.  Likewise,  research  by  Flonour  showed  that  projects  that  devote  15%  to 
20%  of  their  development  budgets  on  SE  exhibit  smaller  cost  and  schedule  variances  [Flonour 
2004]. 

Research  by  Elm  and  others  in  2007  revealed  quantitative  relationships  between  the  application  of 
specific  SE  practices  to  projects,  and  the  performance  of  those  projects  as  measured  by  satisfac¬ 
tion  of  budgets,  schedules,  and  technical  requirements  [Elm  2008]. 

While  this  research  produced  valuable  insights  into  the  role  of  SE  in  development  projects,  these 
studies  were  based  on  a  small  number  of  data  points.  To  better  understand  the  contributions  of  SE 
to  project  success,  in  201 1,  the  National  Defense  Industrial  Association  Systems  Engineering  Di¬ 
vision  (NDIA-SED)  in  collaboration  with  the  Institute  of  Electrical  and  Electronics  Engineers 
Aerospace  and  Electronic  Systems  Society  (lEEE-AESS),  and  the  Software  Engineering  Institute 
(SEI)  of  Carnegie  Mellon  embarked  on  the  Business  Case  for  Systems  Engineering  (BCSE)  pro¬ 
ject.  Initial  results  of  that  project  are  summarized  in  this  report. 

1.2  Purpose 

The  NDIA-SED,  the  lEEE-AESS,  and  the  SEI  are  collaborating  to  expand  and  extend  the  2007 
NDIA/SEI  Systems  Engineering  Effectiveness  Study  [Elm  2008]  to  develop  a  business  case  for 
systems  engineering  (BCSE).  The  mission  of  this  new  study  is  to  assist  the  SE  community  in 
achieving  a  quantifiable  and  persistent  improvement  in  project  performance  through  the  appropri¬ 
ate  application  of  SE  principles  and  practices.  The  primary  steps  in  the  BCSE  process  are 

1.  Identify  SE  principles  and  practices  shown  to  provide  benefit  to  project  performance.  This 
activity  is  an  extension  and  a  confirmation  of  the  prior  NDIA  survey. 

2.  Facilitate  the  adoption  of  the  survey  findings  through  the  development  of  tools,  training,  and 
guidance  for  SE  educators,  system  developers,  and  system  acquirers. 

3.  Establish  an  ongoing  means  of  monitoring  and  tracking  the  impact  of  SE  to  enable  continu¬ 
ous  improvement  of  the  SE  framework  and  the  business  case  for  SE,  thereby  driving  contin¬ 
uous  improvement  of  project  results. 


Gruhl,  W.  “Lessons  Learned,  Cost/Schedule  Assessment  Guide,”  Internal  presentation,  NASA  Comptroller's 
office,  1992., National  Avionics  and  Space  Administration  (NASA),  1992. 
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This  report  addresses  the  results  of  the  first  item.  The  remaining  items  will  be  addressed  by  future 
activities  occurring  within  the  NDIA,  IEEE,  and  the  International  Council  on  Systems  Engineer¬ 
ing  (INCOSE). 
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2  Developing  the  SE  Effectiveness  Survey 


The  first  step  in  building  a  business  case  for  SE  is  to  identify  the  SE  practices  that  have  a  measur¬ 
able  impact  on  project  performance.  This  identification  was  accomplished  via  the  SE  Effective¬ 
ness  Survey,  which  surveyed  system  developers  to  identify  SE  principles  and  practices  applied  to 
development  projects  and  the  performance  achieved  by  those  projects. 

The  SE  Effectiveness  Survey  was  designed  to  test  the  hypothesis 

The  effective  performance  ofSE  best  practices  on  a  development  program  yields  quantifiable 
improvements  in  program  execution  (e.g.,  improved  cost  performance,  schedule  perfor¬ 
mance,  and  technical  performance). 

Test  of  this  hypothesis  requires  both  a  means  of  assessing  the  SE  activities  applied  to  a  project 
and  a  means  of  assessing  the  performance  of  that  project. 

The  process  used  to  develop  this  survey  consisted  of  three  steps: 

1 .  Define  the  survey  population  and  sampling  process. 

2.  Develop  and  test  the  questionnaire. 

3.  Design  the  solicitation  campaign. 

2.1  Defining  the  Survey  Popuiation 

The  first  step  was  to  choose  the  population  to  be  included  in  the  survey.  We  were  interested  in 
reaching  suppliers  of  systems  (i.e.,  products  composed  of  hardware  and/or  software  elements). 
While  the  2007  study  concentrated  solely  on  U.S.  defense  contractors,  a  goal  of  this  survey  was  to 
expand  the  population  to  include  non-defense  projects  and  non-U. S.  system  developers.  As  in  the 
2007  study,  this  survey  focused  on  system  developers  as  opposed  to  system  acquirers  or  service 
suppliers.  This  focus  enabled  us  to  craft  questions  applicable  to  the  majority  of  the  population. 

To  reach  this  population,  we  used  the  resources  of  the  NDIA-SED,  lEEE-AESS,  and  INCOSE,  as 
discussed  in  Section  2.3. 

2.2  Developing  and  Testing  the  Questionnaire 

The  function  of  the  questionnaire  was  to  assess 

•  the  SE  activities  applied  to  individual  projects 

•  the  performance  of  those  projects 

•  factors  other  than  SE  that  could  impact  performance,  such  as 

the  degree  of  challenge  posed  by  the  project 

the  environment  in  which  the  project  was  executed 

prior  experience  of  the  organization  and  the  team  executing  the  project 

The  survey  questions  were  derived  from  those  used  in  the  2007  survey  [Elm  2008]. 
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2.2.1  Assessing  SE  Applied  to  Projects 


To  assess  the  SE  activities  applied  to  the  project,  we  questioned  the  respondents  about  work  prod¬ 
ucts  resulting  from  specific  SE  activities.  In  the  earlier  study,  a  committee  of  SE  experts  chose 
Capability  Maturity  Model  Integration®  (CMMI)  as  a  recognized  standard  that  addresses  key  are¬ 
as  of  SE.  The  committee  reviewed  the  typical  work  products  cited  in  the  then  current  CMMI- 
SE/SW  model  [CMMI  Product  Team  2002]  and  chose  those  work  products  that  they  believed 
were  the  most  representative  of  effective  SE  practices.  We  then  crafted  survey  questions  asking 
about  the  presence  and  characteristics  of  these  work  products.  Most  questions  in  this  section  were 
structured  in  the  form  of  an  assertion  regarding  the  project  being  surveyed: 

This  project  has  a  <work producf>  with  <defined  characteristics> 

where  <work producf>  references  a  typical  CMMI  work  product  identified  for  inclusion 

in  the  survey 
and 

<defined  characteristics>  address  the  contents  of  the  work  product. 

Questions  were  derived  from  other  sources  on  occasion  to  assure  comprehensiveness. 

The  respondent  was  then  asked  to  identify  his  or  her  level  of  agreement  with  this  assertion,  choos¬ 
ing  one  of  the  following:  strongly  disagree,  disagree,  agree,  or  strongly  agree. 

For  this  study,  we  made  only  minor  revisions  to  the  work  product  list  from  the  2007  study.  We 
crafted  82  questions  to  assess  SE  deployment  on  the  project.  These  questions  focused  on  the  pres¬ 
ence  and  the  quality  of  the  51  work  products  listed  in  Table  3. 

Table  3:  Work  Products  Used  to  /Assess  SE  Deployment 


Alternate  solutions 

Baseline  archives 

Baseline  audit  records 

Change  control  board 

Commitment  impacts 

Configuration  baselines 

Configuration  item  list 

Concept  of  operations 

Cost  and  schedule  baselines 

Customer  requirements  list 

Derived  requirements  list 

Earned  Value  Management  System 
(EVMS)  data 

EVMS  updates 

EVMS  variance  thresholds 

Field  problem  assessments 

Field  problem  reports 

Integrated  master  plan 

Integrated  master  schedule 

Interface  control  documents 

Interface  descriptions 

Integrated  product  teams 

Peer  review  plan 

Product  architecture 

Product  integration  process 

Requirements  acceptance  criteria 

Requirements  allocations 

Requirements  approval  process 

Requirements  configuration  records 

Requirements  impact  assessments 

Requirements  management  system 

Requirements  provider  criteria 

Review  of  action  items 

Review  of  issues 

Review  process 

Review  of  selection  criteria 

Risk  list 

Risk  mitigation  plans 

Risk  mitigation  status 

SE  master  schedule 

SE  processes 

SE  tracking  records 

Systems  engineering  master  plan 

Technical  approach 

Trade  study  records 

Use  cases 

Validation  criteria 

Validation  procedures 

Verification  criteria 

Verification  entry  and  exit  criteria 

Verification  procedures 

Work  breakdown  structure 

Capability  Maturity  Model  Integration  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Car¬ 
negie  Mellon  University. 
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These  work  products  were  the  result  of  42  CMMI  standard  practices.  These  practices  were  col¬ 
lected  in  the  12  process  groups  shown  in  Table  4. 


Table  4:  CMMI  Process  Groups 


Requirements  Development 

Project  Planning 

Product  Architecture 

Trade  Studies 

Product  Integration 

Verification 

Validation 

Project  Monitoring  and  Control 

Risk  Management 

Requirements  Management 

Configuration  Management 

Integrated  Product  Team  (IPT)  Based  Capability 

A  more  detailed  list  of  these  work  products  and  their  relationships  to  the  CMMI-SE/SW  model 
are  provided  in  Appendix  A. 

2.2.2  Assessing  Project  Performance 

We  assessed  project  performance  in  terms  of  meeting  schedule,  meeting  budget,  and  satisfying 
technical  requirements.  The  relationship  between  these  three  aspects  of  project  performance  is 
well  known  to  project  managers  as  the  “iron  triangle,”  which  reflects  the  fact  that  a  project  man¬ 
ager  can  often  optimize  the  value  of  one  of  these  parameters,  but  only  at  the  expense  of  the  other 
two.  Thus,  assessment  of  project  performance  demands  attention  to  all  three. 

Reliable  means  of  assessing  project  performance  are  frequently  lacking.  Many  projects  employ 
the  Earned  Value  Management  System  (EVMS),  which  may  seem  to  be  an  ideal  means  of  as¬ 
sessing  project  performance.  However,  our  earlier  study  confirmed  that  the  manner  in  which  it  is 
employed  is  not  consistent.  EVMS  is  calculated  from  variances  from  a  baseline  and  is  therefore 
highly  sensitive  to  revisions  in  that  baseline.  In  some  organizations,  baselines  are  only  revised 
when  responding  to  contract  change  orders.  In  others,  baselines  may  be  changed  during  replan¬ 
ning  activities.  In  yet  others,  baselines  are  revised  at  fixed  intervals.  These  different  baselining 
methods  can  produce  significant  variations  in  the  meaning  of  EVMS  data.  Furthermore,  EVMS 
assesses  the  satisfaction  of  budgetary  and  schedule  needs  only.  It  includes  no  means  of  assessing 
the  satisfaction  of  technical  requirements. 

We  addressed  these  issues  by  collecting  multiple  project  performance  measures  and  looking  for 
the  degree  of  agreement  between  these  measures.  To  maximize  the  availability  of  data  from  the 
participants,  we  used  measures  common  to  many  organizations.  Measures  of  project  performance 
included 

•  EVMS  data  (e.g.,  cost  performance  index  [CPI]  and  schedule  performance  index  [SPI]) 

•  percent  of  requirements  satisfied 

•  changes  in  budget 

•  changes  in  schedule 

•  perception  of  customer  satisfaction 

•  Respondents  are  asked  to  provide  available  data  for  all  relevant  measures. 

An  in-depth  discussion  of  the  collection  and  analysis  of  project  performance  information  is  found 
in  the  report  The  Business  Case  for  Systems  Engineering  Study:  Assessing  Project  Performance 
from  Sparse  Data  [Elm  2012]. 
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2.2.3  Assessing  Other  Factors 


Not  all  projects  are  created  equal;  some  are  more  challenging  than  others.  Challenge  can  arise 
from  technical  considerations  (e.g.,  lack  of  precedent,  immature  technology,  extensive  interopera¬ 
bility  needs,  poor  definition  of  requirements  and  objectives).  It  may  also  arise  from  programmatic 
considerations  (e.g.,  large  size,  long  duration).  Or,  it  may  arise  from  organizational  considerations 
(e.g.,  lack  of  experience,  lack  of  skills).  We  examine  all  of  these  factors  to  form  a  measure  of  pro¬ 
ject  challenge  (PC). 

We  hypothesized  that  the  environment  in  which  the  project  was  executed  could  also  impact  per¬ 
formance.  To  explore  this  hypothesis,  we  collected  information  regarding  the  structure  of  SE 
within  the  organization,  the  end  user  of  the  product,  the  industrial  sector  of  the  organization,  and 
the  country  in  which  the  project  was  executed.  These  factors  were  used  to  better  understand  the 
project  environment  {PE). 

Finally,  we  collected  information  on  the  prior  experience  of  the  organization  and  the  team  execut¬ 
ing  the  project.  This  information  was  combined  into  a  weighted  summed  index  measuring  experi¬ 
ence  {EXP)? 

2.2.4  Testing  the  Survey  Instrument 

We  chose  to  execute  the  survey  online  for  the  respondents’  convenience,  thereby  increasing  the 
likelihood  that  they  would  respond.  To  test  the  questionnaire,  invitations  to  submit  survey  re¬ 
sponses  were  sent  to  members  of  the  SE  Effectiveness  Committee.  They  distributed  the  survey  to 
project  members  within  their  organizations  who  then  submitted  responses  via  the  online  process. 
In  addition  to  submitting  the  responses,  they  also  provided  feedback  on  the  clarity  of  the  questions 
and  the  time  required  to  complete  the  questionnaire.  Most  found  that  they  could  complete  the 
questionnaire  in  30  to  45  minutes — a  time  commitment  that  we  felt  was  acceptable  for  this  study. 

We  collected  the  feedback  from  these  initial  respondents  and  used  it  to  make  minor  improvements 
to  the  survey  instrument.  The  resulting  survey  instrament  is  shown  in  Appendix  B. 

2.3  Designing  the  Solicitation  Campaign 

A  primary  objective  of  the  survey  execution  process  was  to  maximize  the  number  of  qualified 
responses.  This  objective  was  accomplished  in  two  ways: 

•  by  taking  steps  to  maximize  the  response  rate 

•  by  reaching  out  to  a  large  population 

2.3.1  Maximizing  Response  Rate 

We  attempted  to  maximize  the  response  rate  by  making  the  process  of  responding  to  the  survey  as 
convenient  as  possible,  mitigating  concerns  regarding  confidentiality  and  data  security,  and  offer¬ 
ing  incentives  for  responding. 


A  description  of  the  calculation  of  weighted  summed  indices  can  be  found  in  Section  4.1  on  page  10. 
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We  made  the  process  of  responding  to  the  survey  simple  and  convenient  by  providing  the  survey 
online.  To  participate,  a  respondent  merely  had  to  obtain  an  online  account  from  the  survey  serv¬ 
er,  log  in,  and  complete  the  survey. 

Some  organizations  were  expected  to  be  reluctant  to  respond  due  to  the  survey’s  request  for  com¬ 
petition-sensitive  information  identifying  project  performance.  To  mitigate  these  concerns,  we 
collected  all  responses  anonymously.  The  survey  did  not  solicit  information  to  identify  people, 
projects,  or  organizations.  All  data  presented  in  reports  includes  only  aggregate  data  and  does  not 
include  information  traceable  to  any  person,  project,  or  organization. 

Additionally,  we  promised  that  data  would  be  collected  and  handled  by  a  trusted  organization — 
the  SEI,  which  is  a  federally  funded  research  and  development  center  (FFRDC)  that  does  not 
compete  with  any  of  the  responding  organizations  and  is  known  for  handling  sensitive  data.  With 
these  processes,  we  hoped  to  convince  respondents  that  they  could  respond  fully  and  honestly  to 
the  survey  questions  without  fear  of  exposing  critical  information. 

To  encourage  participation,  we  included  an  incentive  to  respond — information.  As  in  the  previous 
study,  results  of  this  survey  provide  a  benchmark  for  SE  performance  among  a  broad  range  of 
system  developers.  Organizations  could  evaluate  themselves  against  this  benchmark  and  develop 
process  improvement  plans  to  obtain  a  competitive  advantage.  Early  access  to  this  benchmark 
information  was  offered  as  a  reward  for  participating  in  the  survey. 

This  report,  made  available  to  the  general  public  immediately  after  completion  of  the  survey,  con¬ 
tains  only  summary  information  of  the  survey  results.  It  does  not  contain  the  distributions  and 
statistical  analyses  of  each  survey  question.  That  information  is  contained  in  a  companion  report. 
The  companion  report  is  available  to  survey  participants  immediately  upon  its  release;  however,  it 
will  not  be  made  available  to  the  broader  public  for  one  year.  Respondents  may  access  this  com¬ 
panion  report  at  https://feedback.sei.cmu.edu/201  l_SE_EffectivenessSurveyResults.htm  using  the 
account  name  and  password  that  they  established  for  submission  of  their  survey  responses. 

An  additional  incentive  was  also  made  available.  Each  submitted  response  was  identified  by  a 
randomly  assigned  account  name  and  a  user-defined  password.  Other  identifying  information 
(e.g.,  respondent  name,  organization,  project)  was  unknown  to  the  researchers.  Flowever,  if  the 
organization  submitting  project  responses  elected  to  subsequently  provide  to  the  SEI  the  account 
names  and  passwords  of  the  responses  submitted  by  its  employees,  the  SEI  could  extract  those 
records  from  the  survey  database  and  perform  an  analysis  similar  to  that  presented  here  but  lim¬ 
ited  to  the  participating  organization’s  own  projects.  This  analysis  would  be  provided  only  to  the 
requesting  organization,  and  would  serve  as  an  organization-specific  baseline  that  could  then  be 
compared  against  the  industry-wide  baseline  derived  from  the  entire  database.  Such  a  comparison 
would  enable  the  requesting  organization  to  identify  its  relative  strengths  and  weaknesses,  and 
create  process  improvement  activities  to  enhance  its  future  performance. 
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2.3.2  Maximizing  Sampie  Size 


Sample  size  was  maximized  by  using  both  a  targeted  and  broadcast  approach  to  reach  potential 
respondents. 

The  targeted  approach  relied  on  networking  through  the  NDIA-SED,  lEEE-AESS,  and  INCOSE. 
Members  of  the  SE  Effectiveness  Committee  are  active  in  all  of  these  organizations.  Furthermore, 
the  memberships  of  these  organizations  often  include  senior  and  mid-level  managers  from  major 
system  design  organizations  worldwide.  By  reaching  out  through  the  NDIA-SED,  lEEE-AESS, 
and  INCOSE,  we  successfully  enlisted  the  leaders  of  many  companies  in  the  SE  Effectiveness 
Committee.  These  committee  members  sponsored  and  promoted  participation  in  the  survey  within 
their  organizations,  soliciting  respondents,  distributing  invitations,  and  expediting  responses. 

The  targeted  approach  was  effective  in  reaching  our  intended  audience  of  the  large  organizations 
that  are  well  represented  within  the  societies  supporting  this  study.  However,  we  also  wished  to 
reach  the  smaller  organizations  participating  in  these  societies.  To  do  this,  we  adopted  a  “broad¬ 
cast”  approach  using  the  resources  of  the  NDIA-SED,  lEEE-AESS,  and  INCOSE  to  reach  their 
respective  memberships.  The  multi-domain,  multi-national  constituency  of  both  the  INCOSE  and 
lEEE-AESS  supported  the  goals  of  reaching  beyond  the  defense  industry  and  beyond  the  collec¬ 
tion  of  U.S.  companies  surveyed  in  the  2007  study. 

Both  the  lEEE-AESS  and  INCOSE  provided  copies  of  their  membership  rosters,  enabling  us  to 
contact  their  members  via  email.  The  NDIA-SED  chose  to  contact  its  members  directly  and  pro¬ 
vide  us  with  their  responses. 

We  had  some  concerns  about  reaching  out  to  the  members  of  these  organizations  and  asking  them 
to  respond  to  the  survey.  The  questionnaire  solicits  information  regarding  projects  executed  with¬ 
in  their  employing  organizations.  We  did  not  wish  to  place  these  individuals  in  a  potential  conflict 
by  asking  them  to  release  information  that  was  not  sanctioned  by  their  companies.  Hence,  our 
initial  contact  via  email,  as  shown  in  Appendix  C,  did  not  ask  them  to  participate  in  the  survey, 
but  merely  asked  them  to  identify  one  or  more  individuals  in  their  organization  who  could  author¬ 
ize  the  organization’s  participation  in  the  survey.  Those  individuals  were  subsequently  invited  to 
participate  in  the  survey. 
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3  Executing  the  Survey 


In  October  201 1,  following  the  targeted  solicitation  approach  discussed  in  Section  2.3,  members 
of  the  SE  Effectiveness  Committee  approached  their  organizations  to  invite  respondents  to  partic¬ 
ipate.  In  some  cases,  the  committee  members  provided  names  and  contact  information  to  the  SEI, 
who  then  sent  the  invitations.  In  other  cases,  the  committee  members  distributed  the  invitations 
themselves  within  their  organizations.  This  approach  resulted  in  the  distribution  of  approximately 
81  invitations  to  participate. 

Also  in  October,  the  SEI  began  to  broadcast  participation  inquires  (see  Appendix  C)  to  the  mem¬ 
berships  of  the  INCOSE  and  lEEE-AESS.  The  NDIA-SED  encountered  some  delays  and  did  not 
broadcast  the  inquiries  to  its  members  until  Febmary  2012.  Results  of  this  process  are  shown  in 
Table  5.  Some  members  of  these  professional  organizations  may  be  from  the  companies  contacted 
via  the  targeted  solicitation  approach. 

Additionally,  some  may  be  members  of  multiple  professional  organizations.  We  cannot  tell  how 
many  such  duplicates  exist  since  we  took  pains  to  keep  competition-sensitive  information  anony¬ 
mous.  No  effort  was  made  to  eliminate  these  duplicative  contacts.  If  anything  the  55%  response 
rate  described  in  Section  5.1  may  be  an  underestimate. 


Table  5:  Participation  Inquiries 


lEEE-AESS 

INCOSE 

NDIA 

TOTAL 

Participation  inquiries  sent 

3,555 

7,756 

... 

11,311 

Participation  inquiries  delivered 

3,341 

6,865 

... 

10,206 

Responses  received 

Referrals  to  others 

11 

85 

6 

102 

Self-referral 

15 

68 

4 

87 

Invitations  sent 

26 

153 

10 

189 

The  invitation  email  contained  a  link  to  the  SETs  survey  web  server.  When  logging  on,  the  re¬ 
spondent  received  a  unique  and  randomly  generated  URL  that  he  or  she  could  use  to  access  a 
copy  of  the  questionnaire.  Access  to  this  secure  site  required  both  knowledge  of  the  URL  and  a 
user-defined  password.  In  this  manner,  only  the  respondent  could  access  his  or  her  assigned  web¬ 
site.  The  respondent  could  then  complete  the  questionnaire  online,  saving  his  or  her  results  incre¬ 
mentally.  At  any  time,  the  respondent  could  exit  the  website  without  losing  the  data  saved.  In  this 
manner,  the  respondent  could  complete  the  questionnaire  over  multiple  sessions.  As  the  respond¬ 
ent  completed  the  questionnaire,  he  or  she  notified  the  survey  server  by  clicking  the  Submit  but¬ 
ton. 

Data  for  the  survey  were  collected  over  a  time  period  extending  from  October  2011  to  March 

2012. 

In  January,  February,  and  March,  reminder  emails  were  sent  to  invitees  encouraging  them  to 
complete  their  submissions  in  a  timely  fashion. 
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4  Analyzing  the  Responses 


The  primary  survey  hypothesis  as  defined  in  Section  2  is  that  SE  best  practices  have  a  quantifiable 
positive  impact  on  project  performance.  We  also  postulated  that  the  degree  of  challenge  imposed 
by  the  project  and  the  environment  in  which  the  project  was  executed  could  also  impact  perfor¬ 
mance.  Mathematically,  we  can  state  this  hypothesis  as 

Perf=  f{PC,  PE,  EXP,  SEC) 

where  Project  challenge  PC 


Project  environment 
Prior  experience 
Systems  engineering  capability 
Project  performance 


PE 


EXP 

SEC 

Perf 


Our  goal  is  to  identify  the  impact  of  PC,  PE,  EXP,  and  SEC  on  Perf.  We  do  this  by  first  scoring 
each  parameter,  and  then  by  identifying  the  relationships  among  them. 

4.1  Scoring 

Measures  for  Perf,  PC,  EXP,  SEC,  and  PE  are  derived  by  combining  the  responses  for  a  set  of 
conceptually  related  questions  into  weighted,  summed  indices  used  as  composite  measures.  There 
are  several  reasons  to  combine  the  component  questions  into  single  composite  indices.  Of  course, 
reducing  the  number  simplifies  visual  interpretation.  While  it  may  seem  counterintuitive,  combin¬ 
ing  the  components  also  follows  a  basic  reliability  principle.  Noise  always  exists  in  survey  data 
(actually  in  measured  data  of  any  kind).  Respondents  can  be  uncertain  about  their  answers  con¬ 
cerning  the  details  of  a  specific  question,  or  the  lack  of  clarity  in  the  wording  of  a  specific  ques¬ 
tion  may  cause  different  respondents  to  attribute  different  meanings  to  the  same  question.  Other 
things  being  equal,  the  unreliability  can  be  averaged  such  that  the  composite  index  is  more  relia¬ 
ble  than  many  or  all  of  its  individual  components  [Guilford  1954,  Coleman  1964,  Hill  2006]. 

Many  of  the  response  categories  range  ordinally  from  “disagree  strongly”  to  “agree  strongly.”  The 
projects’  answers  are  scored  as  1  through  4  respectively  and  then  summed.  Since  the  number  of 
component  items  varies  for  each  composite  measure,  the  scores  are  normalized  to  allow  consistent 
interpretation  of  their  meaning.  Much  like  student  grade  point  averages,  the  composite  scores  are 
divided  by  the  number  of  questions  answered.  The  composite  scores  are  therefore  constrained  to 
range  between  1  and  Calculating  the  composite  scores  this  way  provided  sufficient  variation  to 
enable  meaningful  statistical  comparisons.  Moreover,  the  values  on  the  extremes  of  the  weighted 
summed  indices  require  consistency  of  replies  across  all  of  their  respective  component  questions. 


Such  a  normalization  procedure  is  appropriate  for  ordinal  data  since  the  component  items  fall  in  the  same  con¬ 
strained  range.  Since  the  fractional  differences  cannot  be  interpreted  additively,  the  composite  scores  are  split 
into  two  or  three  groupings  as  appropriate  for  the  data  analysis,  (e.g.,  “Lower,”  “Middle,”  and  “Higher”  group¬ 
ings). 
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See  Appendix  D  for  details  of  the  scoring  process. 

The  project  environment  (P£)  measures  address  factors  other  than  project  challenge  and  SE  capa¬ 
bility  that  could  influence  project  performance.  These  factors  include  the  organization’s  industry 
classification,  percentage  of  project  completion,  SE  organizational  structure,  contract  type,  and  so 
forth.  The  impact  on  project  performance  of  each  of  the  PE  elements  is  evaluated  as  is  the  impact 
of  some  of  them  on  the  relationships  between  SEC  and  Perf. 

4.1 .1  Project  Performance  Analysis 

Project  performance  {Perf)  can  be  measured  and  decomposed  into 


Cost  performance 

{PerfC) 

Schedule  performance 

{PerfS) 

Technical  performance 

{PerfT) 

As  noted  previously,  these  three  factors  are  interrelated,  reflecting  the  fact  that  project  managers 
can  often  optimize  the  value  of  one  of  these  parameters,  but  only  at  the  expense  of  the  other  two. 
For  example,  a  project  manager’s  election  to  reduce  project  cost  can  have  adverse  impacts  on  the 
project  schedule  and  the  achieved  technical  performance  of  the  project.  As  such,  while  looking  for 
relationships  between  SEC  and  the  individual  components  of  Perf  (i.e.,  PerfC,  PerfS,  PerfT)  may 
be  useful,  a  more  complete  picture  is  obtained  by  looking  for  relationships  between  SEC  and  a 
composite  project  performance  variable  combining  all  three  of  these  components. 

PerfC  is  assessed  based  on  evaluations  of  three  factors: 

1 .  estimated  cost  at  completion  (ECAC)  vs.  project  budget 

2.  Earned  Value  Management  System  (EVMS)  Cost  Performance  Index  (CPI) 

3.  perception  of  customer  satisfaction  with  the  cost  performance  of  the  project 

Additional  factors  such  as  project  scope  changes  resulting  from  contract  amendments  and  EVMS 
baseline  management  practices  are  also  considered  in  this  evaluation. 

PerfS  is  assessed  based  on  the  evaluation  of  four  factors: 

1 .  estimated  project  duration  vs.  initial  planned  duration 

2.  EVMS  Schedule  Performance  Index  (SPI) 

3.  deviation  from  approved  schedule 

4.  perception  of  customer  satisfaction  with  the  schedule  performance  of  the  project 

Additional  factors  such  as  project  scope  changes  resulting  from  contract  amendments  and  EVMS 
baseline  management  practices  are  also  considered  in  this  evaluation. 


CMU/SEI-2012-SR-009  |  11 


Perfr  is  assessed  based  on  the  evaluation  of  two  factors: 

1 .  satisfaction  of  system  requirements 

2.  perception  of  customer  satisfaction  with  the  technical  performance  of  the  project 

PerfC,  PerfS,  and  PerfT  are  each  assessed  on  a  scale  ranging  from  1  (very  poor  performance)  to  5 
(very  good  performance).  A  measure  of  overall  project  performance,  Perf,  is  calculated  as  a 
weighted  summed  index  of  these  more  specific  measures.  Details  of  the  development  of  Perf, 
PerfC,  PerfS,  and  PerfT  are  provided  in  Appendix  D  and  in  the  report  The  Business  Case  for  Sys¬ 
tems  Engineering  Study:  Assessing  Project  Performance  from  Sparse  Data  [Elm  2012]. 

4.1 .2  Project  Challenge  (PC) 

The  project  challenge  (PC)  questions  address  a  number  of  diverse  issues  contributing  to  the  diffi¬ 
culty  of  a  project:  issues  such  as  project  size,  project  complexity,  technology  precedents,  and  oth¬ 
ers.  All  of  these  factors  are  combined  into  a  single  PC  measure,  with  the  intent  of  examining  the 
impact  of  project  difficulty  on  Perf  and  the  relationships  between  SEC  and  Perf  The  survey  es¬ 
timated  the  degree  of  challenge  posed  by  the  project  through  a  combination  of  factors  including 

•  sources  of  technical  challenge  •  lack  of  similar  prior  experience 

•  lack  of  well-defined  customer  requirements  •  lack  of  SE  direction  from  the  customer 

•  incomplete  requirements  •  excessive  contract  change  orders 

•  large  contract  value  •  large  change  of  contract  value 

•  long  contract  duration  •  large  change  in  contract  duration 

•  large  project  budget  •  large  change  in  project  budget 

All  factors  were  assessed  and  the  results  were  combined  into  a  weighted  summed  index  to  create 
an  overall  assessment  of  PC  scaled  from  1  (not  very  challenging)  to  4  (very  challenging).  Details 
of  the  development  of  PC  can  be  found  in  Appendix  D. 

4.1.3  Systems  Engineering  Capabiiity  (SEC) 

SEC  is  a  measure  of  the  SE  activities  applied  to  each  project.  In  addition  to  assessing  the  total  SE 
activities  (SEC-Total)  applied  to  the  project,  the  questionnaire  is  designed  to  permit  the  decompo¬ 
sition  of  SEC-Total  into  1 1  measures  of  SE  capability  in  the  identified  process  groups,  as  shown 
in  Table  6. 


Table  6:  SE  Process  Groups 


Requirements  Development  and  Management 

{SEC-REQ) 

Project  Planning 

(SEC-PP) 

Product  Architecture 

[SEC-ARCH) 

Trade  Studies 

[SEC-TRD) 

Product  Integration 

[SEC-PI) 

Verification 

[SEC-VER) 

Validation 

[SEC-VAL) 

Project  Monitoring  and  Control 

[SEC-PMC) 

Risk  Management 

[SEC-RSKM) 

Configuration  Management 

[SEC-CM) 

Integrated  Product  Team  (IPT)  Based  Capability 

[SEC-IPT) 

CMU/SEI-2012-SR-009  |  12 


With  this  decomposition,  it  is  possible  to  look  at  more  specific  correlations  between  these  systems 
engineering  capability  factors  and  project  performance.  Each  of  these  factors  was  assessed  over 
the  range  of  1  (low  capability)  to  4  (high  capability). 

SEC-REQ  assessed  the  requirements  development  and  requirements  management  activities  ap¬ 
plied  to  the  project.  Assessment  factors  included  the  existence  and  characteristics  of  requirements 
lists;  requirements  allocation  documentation;  operational  installation,  maintenance,  and  support 
concept  documentation;  use  cases;  requirements  provider  authorization  criteria;  and  requirements 
impact  analyses. 

SEC-PP  assessed  the  planning  activities  applied  to  the  project.  Assessment  factors  included  the 
existence  and  characteristics  of  SE  planning  processes,  a  work  breakdown  structure  (WBS),  a  pro¬ 
ject  technical  approach,  integrated  master  plan  (IMP),  an  integrated  master  schedule  (IMS),  and  a 
systems  engineering  master  plan  (SEMP). 

SEC-ARCH  assessed  the  product  architecture  activities  applied  to  the  project.  Assessment  factors 
included  the  existence  and  characteristics  of  product  interfaces  and  high-level,  multi-view  product 
structure  documentation. 

SEC-TRD  assessed  the  trade  study  activities  applied  to  the  project.  Assessment  factors  included 
involvement  of  stakeholders  in  trade  studies,  and  the  existence  and  characteristics  of  selection 
criteria  for  alternative  and  trade  study  findings. 

SEC-PI  assessed  the  product  integration  activities  applied  to  the  project.  Assessment  factors  in¬ 
cluded  the  existence  and  characteristics  of  product  integration  processes,  plans,  and  criteria. 

SEC-VER  assessed  the  verification  activities  applied  to  the  project.  Assessment  factors  included 
the  existence  and  characteristics  of  verification  procedures;  acceptance  criteria;  review  processes; 
review  training;  action  item  tracking;  baseline  reviews;  and  non-advocate  reviews. 

SEC-VAL  assessed  the  validation  activities  applied  to  the  project.  Assessment  factors  included 
the  existence  and  characteristics  of  validation  procedures  and  acceptance  criteria. 

SEC-PMC  assessed  the  project  monitoring  and  control  activities  applied  to  the  project.  Assess¬ 
ment  factors  included  the  existence  and  characteristics  of  review  processes,  cost  and  schedule 
baselines,  EVMS  data,  SE  budgets,  and  field  problem  reports. 

assessed  the  risk  management  activities  applied  to  the  project.  Assessment  factors 
included  the  existence  and  characteristics  of  the  risk  management  process  and  risk  mitigation 
plans.  The  integration  of  the  risk  management  process  with  project  cost  projections,  project 
schedule,  and  project  decision  making  was  also  assessed. 

SEC-CM  assessed  the  configuration  management  activities  applied  to  the  project.  Assessment 
factors  included  the  configuration  management  practices  for  requirements  and  baselines;  the  pro¬ 
cess  for  managing  changes  to  controlled  items;  and  the  archiving  of  prior  versions. 

SEC-IPT  assessed  the  use  of  integrated  product  teams  on  the  project.  Assessment  factors  included 
the  presence  of  IPTs  on  the  project,  acquirer  and  supplier  participation  in  IPTs,  the  existence  of  an 
SE-focused  IPX,  and  SE  staff  participation  on  other  IPTs. 
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All  of  the  SE  capabilities  listed  were  also  combined  into  a  weighted  summed  index  measuring 
total  systems  engineering  capability  (SEC-Totaf).  This  index  measure  also  resulted  in  an  assessed 
value  within  the  range  of  1  (low  capability)  to  4  (high  capability). 

Details  of  the  development  of  all  of  these  SE  capability  assessments  are  provided  in  Appendix  D. 

4.1.4  Other  Factors 

Other  factors  with  the  potential  to  impact  project  performance  were  also  assessed,  including  prior 
experience,  contract  type,  SE  organization,  percentage  complete,  and  SE  content,  as  summarized 
in  the  following  sections.  Details  of  the  development  of  all  of  these  assessments  are  provided  in 
Appendix  D. 

4.1. 4.1  Prior  Experience  (EXP) 

We  hypothesized  that  prior  experience  with  similar  projects  within  an  organization  and  within  a 
project  team  could  impact  the  performance  of  the  project.  To  test  this  hypothesis,  we  collected 
data  on  prior  experiences  to  form  an  EXP  score. 

4. 1.4.2  Contract  Type 

We  theorized  that  the  type  of  contract  governing  the  project  could  impact  project  performance. 
While  many  contract  types  may  be  encountered,  we  chose  to  categorize  them  as  either 

fixed  price — ^The  total  contract  value  is  primarily  determined  by  the  initial  contract  (e.g.,  firm 
fixed  price,  fixed  price  with  incentive  fee,  firm  fixed  price — level  of  effort), 
cost-reimbursable — The  total  contract  value  is  primarily  determined  by  the  cost  of  executing  the 
contract  (e.g.,  cost  plus  fixed  fee,  cost  plus  award  fee,  cost  plus  incentive  fee), 
other — The  contract  is  a  type  that  does  not  fit  the  prior  two  categories. 

This  categorization  was  then  used  to  examine  the  impact  of  these  contract  types  on  project  per¬ 
formance. 

4. 1.4. 3  SE  Organization 

Some  organizations  concentrate  the  SE  function  in  a  separate  department.  Others  distribute  SE 
functions  throughout  the  organization.  We  collected  data  to  assess  the  SE  organizational  structure 
and  used  it  to  examine  the  impact  of  this  structure  on  both  SE  deployment  and  project  perfor¬ 
mance. 

4. 1.4.4  Percentage  Complete 

We  collected  data  to  assess  the  degree  of  project  completion  with  the  intent  of  evaluating  when 
SE  activities  were  performed.  Additionally,  we  wished  to  use  this  information  to  assess  confi¬ 
dence  in  the  project  performance  assessments. 

4. 1.4. 5  SE  Content 

We  collected  data  to  assess  the  magnitude  of  the  SE  effort  as  a  percentage  of  the  total  non¬ 
recurring  engineering  effort  with  the  intent  of  identifying  how  the  magnitude  of  this  effort  related 
to  project  performance. 
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4.2  Identifying  Relationships 


Once  the  scores  of  all  parameters  are  calculated  as  discussed  in  Section  4.1,  we  can  begin  identi¬ 
fying  relationships  between  the  scores.  The  primary  objective  of  this  survey  is  to  identify  correla¬ 
tions  between  SEC  and  Perf;  however,  we  are  also  interested  in  the  impacts  of  PC  and  PE  varia¬ 
bles. 

Relationships  are  evaluated  and  presented  in  two  ways:  (1)  via  mosaic  charts  and  (2)  via  non- 
parametric  statistical  analysis.  These  analytic  methods  process  the  data  differently.  Using  multiple 
methods  provides  the  opportunity  to  identify  subtleties  in  the  relationships  and  to  evaluate  corrob¬ 
oration  between  the  methods. 

4.2.1  Mosaic  Charts 

Mosaic  charts  provide  an  intuitive  and  visual  means  of  examining  the  relationship  between  a  de¬ 
pendent  variable  (e.g.,  Perf  depicted  on  the  vertical  axis)  and  an  independent  variable  (e.g.,  SEC- 
Total  or  SEC-PP  depicted  on  the  horizontal  axis). 

Development  of  a  mosaic  chart  starts  with  the  weighted  summed  indices  of  the  independent  vari¬ 
able  (i.e.,  the  score  for  a  process  group  such  as  SEC-PP  or  SEC-RSKM)  and  the  weighted 
summed  indices  of  the  dependent  variable  (i.e.,  Perj).^  For  each  of  the  projects,  the  indices  are 
then  categorized  as  lower,  middle,  or  higher,  based  on  the  establishment  of  breakpoints  that  dis¬ 
tribute  the  projects  evenly  between  these  categories.  The  categorized  scores  are  then  tallied  and 
displayed  on  the  mosaic  chart  showing  the  relationship  between  pairs  of  scores  (e.g.,  SEC-PP  vs. 
Perf,  SEC-RSKM  vs.  Perf. 

As  an  example,  consider  the  notional  mosaic  chart  examining  SEC-PP  vs.  Perf  'm  Figure  4.  To 
develop  this  chart,  we  first  examined  each  project’s  Perf  score  to  assign  it  to  one  of  three  bins — 
one  for  lower  scores,  one  for  middle  scores,  and  one  for  higher  scores.  The  boundaries  for  these 
bins  are  chosen  such  that  the  number  of  projects  in  each  bin  is  approximately  one-third  of  the  total 
sample. 

Next  we  establish  three  similar  groups  based  on  the  project’s  SEC-PP  score  and  assign  each  pro¬ 
ject  to  one  of  three  bins  representing  the  lower,  middle,  and  upper  levels  oi SEC-PP  capability. 
Again,  the  boundaries  for  these  bins  are  chosen  so  that  the  number  of  projects  in  each  bin  is  ap¬ 
proximately  one-third  of  the  total  sample.  These  three  bins  are  identified  across  the  horizontal  axis 
of  the  chart.  Finally,  we  examine  the  projects  in  each  of  the  SEC-PP  bins  to  determine  the  number 
of  projects  that  would  fall  into  each  of  the  three  Perfb'ms,.  These  values  are  displayed  in  the  col¬ 
umns  for  each  of  the  SEC-PP  bins. 


A  description  of  the  calculation  of  weighted  summed  indices  can  be  found  in  Section  4.1  on  page  10. 
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Percentage  of 
prqects  delit'ering 
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of  project 
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W’el  of  project 
performance 
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Figure  4:  Notional  Mosaic  Chart  Key 


The  resulting  chart  is  shown  in  Figure  4:® 


•  Of  the  projects  performing  the  fewest  project  planning  activities 

only  11%  delivered  the  higher  project  performance 
37%  delivered  intermediate  project  performance 
5 1  %  delivered  the  lower  proj ect  performance 

•  Of  the  projects  performing  an  intermediate  amount  of  project  planning  activities 

3 1  %  delivered  the  higher  proj  ect  performance 
36%  delivered  intermediate  project  performance 
33%  delivered  the  lower  project  performance 

•  Of  the  projects  perfonning  the  most  project  planning  activities 

47%  delivered  the  higher  project  performance 
25%  delivered  intermediate  project  performance 
28%  delivered  the  lower  project  performance 

In  this  hypothetical  case,  it  is  evident  that  better  project  planning  capability  is  related  to  better 
project  performance. 

The  mosaic  charts  describe  relative  rather  than  absolute  differences.  The  project  performance  cat¬ 
egories  on  the  vertical  axis  do  not  range  from  worst  possible  performance  score  to  the  best  possi¬ 
ble  performance  score.  Instead,  they  range  from  the  lowest  performance  score  achieved  by  any  of 
projects  in  the  survey  sample  to  the  highest  performance  score  that  was  achieved.  Thus,  on  an 
absolute  scale  of  1  (worst  possible  performance)  to  4  (best  possible  performance),  if  all  of  the  re¬ 
spondents  indicate  that  their  projects  were  performing  relatively  well  and  fell  into  the  range  from 


The  numbers  in  the  chart  are  notional  only.  The  actual  results  for  the  relationship  between  project  planning 
SEC-PP  and  project  performance  Perf  may  be  seen  in  Figure  23  on  page  33.. 
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2  to  4,  the  mosaic  chart  might  consider  those  scoring  from  2  to  2.7  as  “Lower  Performance,”  those 
scoring  from  2.8  to  3.2  as  “Middle  Performance,”  and  those  scoring  from  3.3  to  4  as  “Higher  Per¬ 
formance.”  The  same  is  true  for  the  capability  measure  of  the  horizontal  axis.  It  also  is  relative  in 
nature,  ranging  from  the  lowest  capability  reported  to  the  highest. 

4.2.2  Statistical  Measures 

We  also  use  nonparametric  statistical  analysis  methods  to  examine  these  same  relationships. 

Many  of  the  questions  in  the  questionnaire  are  in  the  form  of  Likert  questions,  structured  as  an 
assertion  to  which  the  respondent  provides  his  degree  of  agreement  (i.e.,  strongly  disagree,  disa¬ 
gree,  agree,  strongly  agree).  Such  responses  are  ordinal  in  nature  because  they  contain  infor¬ 
mation  regarding  rank,  but  not  magnitude.  In  other  words,  we  know  “strongly  agree”  is  better  than 
“agree,”  but  we  do  not  know  how  much  better.  Because  the  majority  of  the  data  are  ordinal,  we 
need  to  use  nonparametric  methods  to  analyze  them. 

For  this  study,  the  relationships  between  the  variables  (e.g.,  SEC-Total  vs.  Perf,  PC  vs.  Perf)  are 
summarized  using  Goodman  and  Kruskal’s  Gamma,  a  measure  of  association  that  expresses  the 
strength  of  relationship  between  two  ordinal  variables.  A  clear,  simple  description  of  Goodman 
and  Kraskal’s  Gamma  appears  in  the  hook  Elementary  Applied  Statistics  by  Linton  Freeman 
[Freeman  1965].  Gamma  is  calculated  by  comparing  pairs  of  the  independent  variable  with  pairs 
of  the  dependent  variable,  as  illustrated  in  Figure  5. 


Input 

Pair  Evaluations 

Pair  Evaluation  Criteria 

data 

(a,,  32)  vs.  (b|,  b2) 

Concordant  nairs  (Pi 

A  B 

(a,,  33)  vs.  (bi,  bj) 

3;  >  aj  and  b;  >  bj 

ai  bi 

(a,,  34)  vs.  (bi,  b4) 

3;  <  aj  and  b;  <  bj 

32  b2 

33  bs 

(a,,  an)  vs.  (bi,  bn) 

Discordant  nairs  tOI 

34  b4 

n(n — 1)/2 

(32,  33)  vs.  (b2,  b3) 

3;  >  aj  and  bi  <  bj 

pairs 

(32,  34)  vs.  (bj,  b4) 

3;  <  aj  and  b;  >  bj 

Sn-l  bn-1 

(32,  a„)  vs.  (b2,  bn) 

an  b„ 

Discarded  nairs 

(a„-l,  an)  vs.  (bn-1,  bn) 

31  =  aj  or  bj  =  bj 

Figure  5:  Gamma  Calculation 

Gamma  is  computed  as  (P-Q)/(P+Q);  in  other  words,  the  excess  of  concordant  pairs  as  a  percent¬ 
age  of  all  pairs,  ignoring  ties.  Similar  to  Pearson’s  product  moment  correlation  coefficient  (r). 
Gamma  varies  from  (+1)  to  (-1),  with 

•  values  near  -1  indicating  a  strong  opposing  relationship 

•  values  near  0  indicating  a  weak  or  no  relationship  (statistical  independence) 

•  values  near  +1  indicating  a  strong  supporting  relationship 

Gamma  is  a  proportional  reduction  in  error  (PRE)  statistic,  so  understanding  its  value  is  intuitive¬ 
ly  straightforward.  Conceptually  similar  to  Pearson’s  for  interval  or  ratio  data,  the  value  of 
gamma  is  the  proportion  of  paired  comparisons  where  knowing  the  rank  order  of  one  variable 
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reduces  the  proportionate  error  in  predicting  the  rank  order  of  the  other  variable.  For  example,  if 
gamma  is  0.65  then  knowing  the  rank  of  the  independent  variable  reduces  the  error  in  predicting 
the  rank  of  the  dependent  variable  by  65  percent.  In  this  sense,  gamma  is  a  measure  of  relative 
accuracy. 

The  calculation  of  Gamma  is  not  performed  using  the  raw  indices  for  the  independent  and  de¬ 
pendent  variables.  Instead,  it  is  performed  using  the  indices  after  categorization  as  lower,  middle, 
or  higher,  as  used  in  the  mosaic  charts.  The  Gamma  calculation  simply  pairs  the  responses  of  the 
categorized  indices  and  counts  the  concordant  and  discordant  pairs. 

Notionally,  Gamma  values  may  be  interpreted  as 


0  <  I  Gamma  \  <  0.2 
0.2  <  I  Gamma  \  <  0.3 
0.3  <  I  Gamma  \  <  0.4 
0.4  <  I  Gamma  \ 


Weak  relationship 
Moderate  relationship 
Strong  relationship 
Very  strong  relationship 


The  mosaics  that  appear  beginning  in  Section  5.3  also  display  p-values  from  statistical  tests  asso¬ 
ciated  with  each  Gamma  statistic.  No  statistical  relationship  can  ever  be  fully  corroborafed.  How¬ 
ever  we  can  esfimate  the  probability  that  an  observed  relationship  is  likely  to  occur  by  chance 
alone.  The  lower  the  p-value,  the  less  likely  the  magnitude  of  the  relationship  is  to  be  a  chance 
occurrence.  By  convention,  values  ofp<  0.05  or p<  0.01  typically  are  used  as  a  basis  for  reject¬ 
ing  the  null  hypothesis  (i.e.,  having  confidence  that  the  relationship  is  not  specious). 

Because  of  the  small  number  of  cases  in  the  present  survey,  the  p-values  for  some  of  the  weaker 
relationships  are  greater  than  0.05.  However,  the  mosaic  charts  and  related  Gamma  values  them¬ 
selves  are  more  meaningful  for  understanding  the  results  than  are  the  p-values  per  se. 

Given  the  way  in  which  the  sample  was  drawn,  we  cannot  generalize  our  univariate  findings  to 
the  larger  population  of  DoD  programs.  The  distribution  of  scores  for  any  single  variable  may 
differ  from  other  such  programs;  however,  there  is  sufficient  variation  to  analyze  the  relationships 
among  the  variables.  It  is  those  relationships  that  allow  us  to  address  the  validity  of  assertions 
about  the  effects  systems  engineering  activities  have  on  program  performance  under  varying  cir¬ 
cumstances. 
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5  Survey  Results 


This  section  examines  the  results  of  the  analysis  of  survey  responses.  It  contains  only  summaries 
of  aggregated  information  derived  from  survey  results. 

Detailed  information  showing  the  distributions  and  statistical  analyses  of  each  survey  question  are 
contained  in  a  companion  report,  The  Business  Case  for  Systems  Engineering  Study:  Detailed 
Response  Data  [Elm  2013].  This  detailed  information  is  useful  in  defining  a  benchmark  against 
which  system  developers  can  compare  their  SE  capabilities  to  manage  SE  process  improvements. 
As  a  reward  for  their  participation,  the  companion  report  is  also  available  with  the  publication  of 
this  report  to  all  those  who  submitted  useable  responses  to  the  SE  Effectiveness  Survey.^  The 
companion  report  will  be  made  available  to  the  general  public  one  year  later. 

5.1  Response  and  Response  Rate 

Invitations  were  delivered  to  270  potential  participants — 81  resulting  from  targeting  specific  or¬ 
ganizations  and  189  from  inquiries  broadcast  through  the  NDIA,  IEEE,  and  INCOSE.  There  most 
probably  were  duplicates  between  the  invitations  sent  as  a  result  of  the  inquiries  and  the  8 1  sent 
through  the  auspices  of  the  SE  Effectiveness  Committee  members.  We  cannot  tell  how  many  such 
duplicates  exist  since  we  took  pains  to  keep  competition-sensitive  information  anonymous.  Thus 
it  is  likely  that  the  number  of  invitees  is  less  than  270. 

These  invitations  resulted  in  the  receipt  of  148  responses  that  were  sufficiently  complete  to  sup¬ 
port  analysis.*  This  number  of  responses  amounts  to  an  effective  response  rate  of  at  least  55%. 

Selection  biases  are  a  concern  in  any  survey  when  the  sample  is  not  based  on  random,  known 
probability  of  selection  criteria.  Due  to  the  means  available  to  us  for  soliciting  respondents,  we 
could  not  ensure  that  the  participation  inquiries  were  selected  by  random  criteria.  However,  we 
did  take  steps  to: 

•  Ensure  anonymity  to  all  of  the  survey  respondents  to  maximize  the  likelihood  of  trath- 
fulness 

•  Stress  the  need  to  provide  candid  responses  to  make  the  results  useful  to  themselves  as 
well  as  others. 

•  Provide  detailed  instructions  with  criteria  to  help  corporate  management  from  the  target¬ 
ed  organizations  randomize  their  selections. 


Survey  participants  can  use  the  URL  and  password  that  they  used  to  submit  their  response  to  the  SE  Effective¬ 
ness  Survey  to  access  the  SEI  website  at 

https://feedback.sei.cmu.edu/201 1_SE_EffectivenessSurveyResults.htm  and  download  the  companion  report. 
As  you  may  notice  beginning  in  Section  5.4,  a  few  of  the  respondents  did  not  answer  all  of  the  questions. 
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In  the  absence  of  required,  auditable  benchmarks  we  cannot  generalize  elsewhere  the  distributions 
of  scores  on  our  individual  measures  of  program  performance  or  implementation  of  systems  engi¬ 
neering  practices  when  each  of  them  is  considered  alone.  However  there  is  sufficient  covariation 
in  the  relationships  between  program  performance  and  the  systems  engineering  measures  to  draw 
useful  conclusions  about  the  impact  of  those  practices  on  program  outcomes. 

Despite  our  efforts  to  reach  out  to  non-defense  and  non-U. S.  system  developers,  the  majority  of 
the  responses  came  from  U.S.  defense  industry  organizations  that  were  executing  contracts  within 
the  U.S.  for  the  U.S.  DoD.  However,  as  shown  in  Figure  6,  21%  of  survey  participants  are  not 
defense  manufacturing  or  service  providers.^  As  shown  in  Figure  7,  22%  of  projects’  end  users  are 
not  from  the  defense  industry.  As  shown  in  Figure  8,  88%  of  the  development  engineering  was 
or  will  be  done  in  the  United  States. 


The  categories  in  Figure  5  are  Standard  Industrial  Classification  (SIC)  codes. 

The  categories  in  Figure  6  were  established  after  considerable  discussion  by  members  of  the  Systems  Engi¬ 
neering  Effectiveness  working  group. 
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Which  of  these  best  describes  your  industry  or 
service? 
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Figure  6:  Industry 


Which  of  the  following  best  describes  the  ultimate 
end-user  of  this  product? 
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Figure  7:  System  End  Users 
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Figure  8:  Country  of  Project  Execution 


We  also  collected  information  regarding  the  size  of  the  responding  projects,  as  measured  by  their 
contract  value  (Figure  9).  The  contract  values  are  expressed  exponentially  on  the  horizontal  axis 
because  of  the  extremely  wide  variation  across  the  projects.  The  median  contract  value  is  $50.5 
million.  The  arithmetic  mean  is  much  larger  ($488  million)  because  of  the  extremely  high  contract 
values  above  the  median. 


5.2  Project  Performance 

As  noted  in  Section  4.1.1,  total  project  performance  {Perf)  comprises  cost  performance  (PerfC), 
schedule  performance  (PerfS),  and  technical  performance  {PerfT). 

Distribution  of  Perf  for  the  surveyed  sample  is  shown  in  Figure  10. 
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Istquartile  3.03 
Median  (2nd  quartile)  3.58 


Figure  1 0:  Perf  Response  Distribution 


A  Perf  value  of  3.00  may  be  interpreted  as  a  project  that  is  on  schedule,  on  budget,  and  satisfying 
technical  requirements.  Values  greater  than  three  indicate  superior  performance  (e.g.,  under  budg¬ 
et,  ahead  of  schedule,  and/or  exceeding  technical  requirements),  while  values  less  than  three  rep¬ 
resent  inferior  performance. 

The  median  value  of  3.58  indicates  that  the  sample  of  projects  surveyed,  in  the  aggregate,  exceed¬ 
ed  performance  expectations  in  terms  of  cost,  schedule,  and/or  technical  performance. 

The  high  Perf  scores  reported  here  in  Section  5.2  cannot  be  generalized  from  this  sample  to  all 
DoD  programs.  However,  it  is  the  relationships  between  Perf  and  the  other  variables  discussed 
throughout  all  of  Section  5  that  allow  us  to  judge  the  extent  to  which  systems  engineering  activi¬ 
ties  can  affect  program  performance.  Whether  or  not  the  sample  is  biased  does  not  matter  as  long 
as  any  such  bias  is  consistent  across  the  projects  and  programs  surveyed,  regardless  of  their  values 
on  the  independent  variables. 

Distribution  of  the  cost  performance  (PerfC)  for  the  surveyed  sample  is  shown  in  Figure  11. 
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Again,  a  PerfC  value  of  3.00  may  be  interpreted  as  a  project  that  is  expected  to  execute  within  ± 
5%  of  budget.  Values  greater  than  3  indicate  under-budget  performance  while  values  less  than  3 
represent  over-budget  performance. 

The  median  value  of  3.50  indicates  that  the  sample  of  projects  surveyed,  in  the  aggregate,  exceed¬ 
ed  cost  performance  expectations. 

Distribution  of  schedule  performance  (PerfS)  for  the  surveyed  sample  is  shown  in  Figure  12. 


Again,  a  PerfS  value  of  3.00  may  be  interpreted  as  a  project  that  is  expected  to  execute  within  ± 
5%  of  the  planned  duration.  Values  greater  than  3  indicate  accelerated  performance,  while  values 
less  than  3  represent  delayed  performance. 

The  median  value  of  3.58  indicates  that  the  sample  of  projects  surveyed,  in  the  aggregate,  exceed¬ 
ed  schedule  performance  expectations. 

Distribution  of  the  technical  performance  (PerfT)  for  the  surveyed  sample  is  shown  in  Figure  13. 

Istquartile  3.00 
Median  (2nd  quartile)  3.67 


Figure  13:  PerfT  Response  Distribution 


CMU/SEI-2012-SR-009  |  24 


Again,  a  PerfT  value  of  3.00  may  be  interpreted  as  a  project  that  is  on  track  to  meet  most  of  its 
technical  requirements.  Values  greater  than  3  indicate  technical  performance  exceeding  expecta¬ 
tions  while  values  less  than  3  represent  technical  performance  falling  short  of  expectations. 

The  median  value  of  3.67  indicates  that  the  sample  of  projects  surveyed,  in  the  aggregate,  exceed¬ 
ed  technical  expectations. 

An  examination  of  the  spread  of  these  performance  measures  is  also  instractive.  Among  the  three 
components  of  performance,  PerfT  has,  the  narrowest  distribution,  indicating  that  projects  meet 
their  technical  expectations  (as  measured  by  this  survey)  with  good  consistency.  PerfS  has  a  wid¬ 
er  distribution,  indicating  that  projects  are  less  consistent  at  controlling  schedule  performance. 
PerfChas  the  widest  distribution,  revealing  the  least  consistency  in  managing  project  costs. 

Perf  'is  used  as  the  primary  variable  for  comparison  with  SEC  variables.  As  discussed  in  Section 
4.2.1,  the  first  step  in  preparing  the  mosaic  charts  showing  these  relationships  is  to  divide  the  re¬ 
sponses  into  three  groups  based  on  Perf.  This  activity  is  done  by  examining  the  cumulative  distri¬ 
bution  function  of  Perf  (Figure  14)  to  identify  two  breakpoints  that  distribute  the  projects  evenly 
into  three  bins." 


Examining  the  data,  we  can  identify  these  breakpoints  at  3.10  and  3.59.  Thus,  for  all  future  analy¬ 
sis,  we  categorize  projects  as  follows: 

1  <  Perf  <  3.30  =>  Lower  project  performance 

3.30  <  Perf  <  3.85  ^  Middle  project  performance 

3.85  <  Perf  ^  Higher  project  performance 

5.3  Project  Challenge 

The  distribution  of  the  responses  that  assessed  project  challenge  {PC)  is  shown  in  Figure  15  with 
the  value  of  1  representing  very  low  challenge  and  4  representing  very  high  challenge. 


”  More  detail  about  establishing  the  cutting  points  can  be  found  in  The  Business  Case  for  Systems  Engineering 
Study:  Assessing  Project  Performance  from  Sparse  Data  [Elm  2012], 
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Istquartile  2.22 
Median  (2nd  quartile)  2.50 


Figure  1 5:  PC  Response  Distribution 

The  median  of  2.50  indicates  that  the  sampled  projects  were  nearly  in  the  middle  of  the  continuum 
of  project  challenge. 

As  part  of  preparing  the  mosaic  chart  showing  the  relationship  between  PC  and  Perf,  three  groups 
for  PC  were  established  with  breakpoints  at  2.33  and  2.65.  These  breakpoints  resulted  in  47  pro¬ 
jects  categorized  as  presenting  lower  challenge,  54  as  presenting  middle  challenge,  and  47  as  pre¬ 
senting  higher  challenge.  The  resulting  mosaic  chart  is  shown  in  Figure  16. 


Examining  this  chart  reveals  a  moderately  negative  relationship  between  PC  and  Perf.  This  rela¬ 
tionship  is  consistent  with  intuition — you  would  expect  more  challenging  projects  to  have  greater 
difficulty  achieving  performance  expectations.  The  percentage  of  projects  delivering  higher  per¬ 
formance  decreased  from  36%  to  33%  to  28%  as  PC  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  increased  from  17%  to  33% 
to  49%  as  PC  increased.  This  relationship  is  characterized  by  a  moderate  Gamma  value  of  -0.26 
and  a  p-value  of  0.015. 

5.4  Systems  Engineering  Capabiiities 

The  following  12  sections  discuss  the  results  of  the  survey  for  systems  engineering  capabilities. 

5.4.1  Totai  Systems  Engineering  Capabiiity 

As  discussed  previously,  the  total  systems  engineering  capability  (SEC-Totaf)  applied  to  a  project 
is  assessed  as  the  amalgam  of  SE  capabilities  applied  in  the  following  process  groups: 

Requirements  development  and  management  (SEC-RE Q) 

Project  planning  (SEC-PP) 

Product  architecture  (SEC-ARCH) 

Trade  studies  (SEC-TRD) 

Product  integration  (SEC-PI) 

Verification  (SEC-VER) 

Validation  (SEC-VAL) 

Project  monitoring  and  control  (SEC-PMC) 

Risk  management  (SEC-RSKM) 

Configuration  management  (SEC-CM) 

Integrated  product  team  based  capability  (SEC-IPT) 

The  distribution  of  SEC-Total  is  shown  in  Figure  17,  with  a  value  of  1  representing  very  poor  SE 
deployment  and  4  representing  very  good  SE  deployment. 

Istquartile  2.78 
Median  (2nd  quartile)  3.03 


Figure  17:  SEC-Total  Response  Distribution 

The  median  of  3.03  indicates  that  the  typical  project  in  the  sample  offers  good  deployment  of  SE, 
with  some  room  for  improvement. 
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In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-Total  and  Perf,  three  groups 
ior  SEC-Total  were  established  with  breakpoints  at  2.85  and  3.27.  This  resulted  in  48  projects 
categorized  as  having  Xo'nqv  SEC-Total  capability,  49  as  having  middle  SEC-Total  capability,  and 
51  as  having  higher  SEC-Total  capability.  These  breakpoints  result  in  the  mosaic  chart  shown  in 
Figure  18. 
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Gamma  =  0.49  p-value  <  0.001 


Figure  18:  SEC-Total  vs.  Perf 

Examination  of  this  chart  reveals  a  very  strong  supporting  relationship  between  SEC-Total  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  from  15%  to  24%  to 
57%  as  SEC-Total  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  52%  to 
29%  to  20%  as  SEC-Total  increased.  This  relationship  is  characterized  by  a  Gamma  value  of 
+0.49  and  a  very  low  p-value  less  than  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  project  challenge  {PC).  The 
chart  on  the  left  side  of  Figure  1 9  shows  the  relationship  between  SEC-Total  and  Perf  for  those 
projects  with  lower  PC  {PC  <  2.45).  This  set  contains  73  projects.  Of  these,  22  were  assessed  as 
deploying  lower  SEC-Total  capabilities,  26  deploying  middle  SEC-Total  capabilities,  and  25  de¬ 
ploying  higher  SEC-Total  capabilities. 

The  chart  shows  a  strong  supporting  relationship  between  SEC-Total  and  Perf  with  the  percent¬ 
age  of  projects  delivering  higher  performance  changing  from  23  %  to  23  %  to  52  %  as  SEC-Total 
increased  from  lower  to  middle  to  higher.  Similarly,  the  percentage  of  projects  delivering  lower 
performance  decreased  from  32%  to  19%  to  12%  as  SEC-Total  increased. 
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Thus,  for  the  lower  challenge  projects,  the  likelihood  of  delivering  higher  performance  more  than 
doubled  with  mvproyQd  SEC-Total,  while  those  delivering  lower  performance  was  reduced  to  less 
than  half.  This  relationship  is  characterized  by  a  Gamma  value  of  +0.34  and  a  low  p-value  of 
0.029. 


Low  PC 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=22)  (n=26)  (n=25) 


Gamma  =  0.34  p-value  =  0.029 


Figure  19:  SEC-Total  vs.  Perf  Controlled  by  PC 


High  PC 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=26)  (n=23)  (n=26) 


Gamma  =  0.62  p-value  <  0.001 


The  chart  on  the  right  side  of  Figure  19  shows  the  relationship  between  SEC-Total  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  26  were  as¬ 
sessed  as  deploying  lower  SEC-Total  capabilities,  23  deploying  middle  SEC-Total  capabilities, 
and  26  deploying  higher  SEC-Total  capabilities. 


The  chart  shows  a  very  strong  supporting  relationship  between  SEC-Total  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  8%  to  26%  to  62%  as  SEC- 
Total  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  69%  to 
39%  to  27%  as  SEC-Total  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  more  than  sevenfold  and  that  of  delivering  lower  per¬ 
formance  decreased  by  almost  two-thirds  with  improved  SEC-Total.  This  relationship  is  charac¬ 
terized  by  a  Gamma  value  of  +0.62  and  a  very  low  p-value  less  than  0.001. 


These  findings  support  the  concept  that  projects  of  all  complexity  benefit  from  stronger  systems 
engineering  practices,  but  the  impacts  are  even  greater  for  challenging  projects. 


5.4.2  Requirements  Development  and  Management 


The  distribution  of  the  responses  assessing  the  requirements  development  and  management  activi¬ 
ties  {SEC-REQ)  of  the  projects  is  shown  in  Figure  20,  with  a  value  of  1  representing  very  poor 
requirements  development  and  management  and  4  representing  very  good  requirements  develop¬ 
ment  and  management. 
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1st  quartile  2.79 
Median  (2nd  quartile)  3.15 


Figure  20:  SEC-REQ  Response  Distribution 

The  median  of  3.15  indicates  good  application  of  requirements  development  and  management 
best  practices,  relative  to  the  survey  questions,  but  still  with  some  room  for  improvement. 


In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-REQ  and  Perf,  three  groups 
for  SEC-REQ  were  established  with  breakpoints  at  2.90  and  3.45.  These  breakpoints  resulted  in 
48  projects  categorized  as  having  \owQr  SEC-REQ  capability,  50  as  having  middle  SEC-REQ 
capability,  and  50  as  having  SEC-REQ  capability.  The  resulting  mosaic  chart  is  shown  in 
Figure  21. 
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Gamma  =  0.44  p-value  <  0.001 


Figure  21:  SEC-REQ  vs.  Perf 


Examination  of  this  chart  reveals  a  very  strong  supporting  relationship  between  SEC-REQ  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  changed  from  21%  to  18%  to  58% 
as  SEC-REQ  increased  from  lower  to  middle  to  higher. 


CMU/SEI-2012-SR-009  |  30 


Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  50%  to 
30%  to  20%  as  SEC-REQ  increased.  This  relationship  is  characterized  by  a  very  strong  Gamma 
value  of  +0.44  and  a  very  low  p-value  less  than  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  22  shows  the  relationship  between  SEC-REQ  and  Perf  for  those  projects  with  low¬ 
er  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  20  were  assessed  as  deploying  lower 
SEC-REQ  capabilities,  26  deploying  middle  SEC-REQ  capabilities,  and  27  deploying  higher 
SEC-REQ  capabilities. 

The  chart  shows  a  strong  supporting  relationship  between  SEC-REQ  and  Perf,  with  the  percent¬ 
age  of  projects  delivering  higher  performance  changing  from  25%  to  15%  to  56%  as  SEC-REQ 
increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  35%  to  15% 
to  15%  as  SEC-REQ  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliv¬ 
ering  higher  performance  more  than  doubled  with  improved  SEC-REQ,  and  that  of  delivering 
lower  performance  was  reduced  by  57%.  This  relationship  is  characterized  by  a  strong  Gamma 
value  of  +0.36  and  a  low  p-value  of  0.017. 


Figure  22:  SEC-REQ  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  22  shows  the  relationship  between  SEC-REQ  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  28  were  as¬ 
sessed  as  deploying  \owev  SEC-REQ  capabilities,  24  deploying  middle  SEC-REQ  capabilities, 
and  23  deploying  higher  SEC-REQ  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-REQ  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  18%  to  21%  to  61%  as  SEC- 
REQ  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  61%  to 
46%  to  26%  as  SEC-REQ  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  more  than  tripled,  and  that  of  delivering  lower  performance  was 
reduced  by  57%  with  impmyed  SEC-REQ.  This  relationship  is  characterized  by  a  very  strong 
Gaimna  value  of  +0.50  and  a  very  low  p-value  of  0.001. 

We  infer  that  a  higher  requirements  development  and  management  capability  is  strongly  associat¬ 
ed  with  better  program  performance,  particularly  on  challenging  projects.  This  relationship  sup¬ 
ports  our  intuitive  impressions  that  projects  have  a  better  chance  of  success  if  they  are  able  to  ef¬ 
fectively  manage  the  project  scope  and  changes  to  the  requirements  baseline. 

5.4.3  Project  Planning 


The  distribution  of  the  responses  assessing  project  planning  activities  (SEC-PP)  of  the  projects  is 
shown  in  Figure  23,  with  a  value  of  1  representing  very  poor  project  planning  and  4  representing 
very  good  project  planning. 
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Figure  23:  SEC-PP  Response  Distribution 


The  median  of  2.98  indicates  moderate  application  of  project  planning  best  practices,  with  room 
for  improvement. 


In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-PP  and  Perf,  three  groups 
for  SEC-PP  were  established  with  breakpoints  at  2.82  and  3.25.  These  breakpoints  resulted  in  48 
projects  categorized  as  having  lower  SEC-PP  capability,  50  as  having  middle  SEC-PP  capability, 
and  50  as  having  higher  SEC-PP  capability.  The  resulting  mosaic  chart  is  shown  in  Figure  25. 
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Figure  24:  SEC-PP  i/s.  Perf 


Examination  of  this  chart  reveals  a  very  strong  supporting  relationship  between  SEC-PP  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  from  13%  to  34%  to 
50%  as  SEC-PP  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  54%  to 
24%  to  22%  as  SEC-PP  increased.  This  relationship  is  characterized  by  a  very  strong  Gamma 
value  of  +0.46  and  a  very  low  p-value  less  than  0.001,  suggesting  high  confidence  in  the  relation¬ 
ship  between  SEC-PP  and  Perf. 


We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  25  shows  the  relationship  between  SEC-PP  and  Perf  for  those  projects  with  lower 
PC  {PC  <  2.45).  This  set  contains  73  projects.  Of  these,  17  were  assessed  as  deploying  lower 
SEC-PP  capabilities,  27  deploying  middle  SEC-PP  capabilities,  and  29  deploying  higher  SEC- 
PP  capabilities. 

The  chart  shows  a  weak  supporting  relationship  between  SEC-PP  and  Perf  with  the  percentage 
of  projects  delivering  higher  performance  increasing  from  18%  to  37%  to  38%  as  SEC-PP  in¬ 
creased  from  lower  to  middle  to  higher. 

Similarly,  the  percentage  of  projects  delivering  lower  performance  changed  from  24%  to  19%  to 
21%  as  SEC-PP  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  delivering 
higher  performance  more  than  doubled  with  improved  SEC-PP,  while  the  likelihood  of  delivering 
lower  performance  remained  substantially  unchanged. 

This  relationship  is  characterized  by  a  weak  Gamma  value  of  +0. 16  and  a  high  p-value  of  0.3 13, 
indicating  that  it  is  difficult  to  draw  any  reliable  conclusions  from  this  sample.  SEC-PP  does  not 
appear  to  have  a  significant  correlation  with  the  performance  of  less  challenging  projects. 
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Figure  25:  SEC-PP  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  25  shows  the  relationship  between  5'£'C-PP  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  3 1  were  as¬ 
sessed  as  deploying  lower  5'£'C-PP  capabilities,  23  deploying  middle  5'£'C-PP  capabilities,  and  21 
deploying  higher  SEC-PP  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-PP  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  10%  to  30%  to  67%  as  SEC- 
PP  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  71%  to 
30%  to  24%  as  SEC-PP  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  almost  sevenfold,  and  that  of  delivering  lower  perfor¬ 
mance  decreased  by  nearly  two-thirds  with  improved  SEC-PP.  This  relationship  is  characterized 
by  a  very  strong  Gamma  value  of  +0.65  and  a  very  low  p-value  less  than  0.001. 

We  can  infer  that  higher  SEC-PP  capability  is  strongly  correlated  with  project  performance  on 
challenging  projects  where  planning  may  be  even  a  more  critical  need  due  to  project  size  and 
complexity. 

5.4.4  Product  Architecture 

The  distribution  of  the  responses  assessing  product  architecture  activities  {SEC-ARCH)  of  the 
projects  is  shown  in  Figure  26  with  a  value  of  1  representing  very  poor  product  architecture  work 
and  4  representing  very  good  product  architecture  work. 
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Istquartile  2.80 
Median  (2nd  quartile)  3.20 


Figure  26:  SEC-ARCH  Response  Distribution 

The  median  of  3.20  indicates  good  application  of  product  architecture  best  practices,  but  still  with 
some  room  for  improvement. 


In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-ARCH  and  Perf  three 
groups  for  SEC-ARCH  were  established  with  breakpoints  at  2.90  and  3.40.  These  breakpoints 
resulted  in  45  projects  categorized  as  having  lower  capability,  54  as  having  middle 

SEC-ARCH  capability,  and  49  as  having  SEC-ARCH  capability.  The  resulting  mosaic 

chart  is  shown  in  Figure  27. 
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Figure  27:  SEC-ARCH  vs.  Perf 


Examination  of  this  chart  reveals  a  very  strong  supporting  relationship  between  SEC-ARCH  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  from  16%  to  3 1%  to 
49%  as  SEC-ARCH  increased  from  lower  to  middle  to  higher. 


CMU/SEI-2012-SR-009  |  35 


Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  49%  to 
33%  to  18%  as  SEC-ARCH  increased.  This  relationship  is  characterized  by  a  very  strong  Gamma 
value  of  +0.41  and  a  very  low  p-value  less  than  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  28  shows  the  relationship  hQtmQQn  SEC-ARCH  and  Pej/ for  those  projects  with 
lower  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  21  were  assessed  as  deploying  low¬ 
er  i'CC-zlPC/F  capabilities,  28  deploying  middle  SEC-ARCH  capabilities,  and  24  deploying 
higher  SEC-ARCH  capabilities. 


The  chart  shows  a  strong  supporting  relationship  between  SEC-ARCH  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  24%  to  29%  to  46%  as  SEC- 
ARCH  increased  from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  33%  to 
21%  to  8%  as  SEC-ARCH  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of 
delivering  higher  performance  nearly  doubled  with  improved  SEC-ARCH,  and  that  of  delivering 
lower  performance  was  reduced  by  over  75%.  This  relationship  is  characterized  by  a  strong 
Gamma  value  of  +0.3 1  and  a  p-value  of  0.051. 
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Figure  28:  SEC-ARCH  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  28  shows  the  relationship  between  SEC-ARCH  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  24  were  as¬ 
sessed  as  deploying  lov/Qr  SEC-ARCH  capabilities,  26  deploying  middle  SEC-ARCH  capabili¬ 
ties,  and  25  deploying  higher  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-ARCH  and  Perf,  with  the 
percentage  of  projects  delivering  higher  performance  increasing  from  8%  to  35%  to  52%  as  SEC- 
ARCH  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  63%  to 
46%  to  28%  as  SEC-ARCH  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  over  sixfold,  and  that  of  delivering  lower  performance 
was  reduced  by  more  than  half  with  improved  SEC-ARCH.  This  relationship  is  characterized  by  a 
very  strong  Gamma  value  of  +0.49  and  a  very  low  p-value  of  0.001. 

We  infer  that  stronger  product  architecture  capability  is  associated  with  better  performance  on 
projects  of  all  types,  but  the  benefits  are  particularly  evident  on  challenging  projects  where  size 
and  rework  of  architectural  problems  could  be  significant  issues. 

5.4.5  Trade  Studies 

The  distribution  of  the  responses  assessing  trade  study  activities  {SEC-TRD)  of  the  projects  is 
shown  in  Figure  29  with  a  value  of  1  representing  very  poor  trade  study  work  and  4  representing 
very  good  trade  study  work. 

Istquartile  2.67 
Median  (2nd  quartile)  3.00 


Figure  29:  SEC-TRD  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  trade  study  best  practices,  with  room  for 
improvement. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-TRD  and  Perf,  three  groups 
for  SEC-TRD  were  established  with  breakpoints  at  2.80  and  3.20.  These  breakpoints  resulted  in 
46  projects  categorized  as  having  \ovrQr  SEC-TRD  capability,  58  as  having  middle  SEC-TRD 
capability,  and  44  as  having  higher  SEC-TRD  capability.  The  resulting  mosaic  chart  is  shown  in 
Figure  30. 


CMU/SEI-2012-SR-009  |  37 


Higher 

Perf 


Middle 

Perf 


Lower 

Perf 


All 


Gamma  =  0.38  p-value  <  0.001 


Figure  30:  SEC-TRD  vs.  Perf 


Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  SEC-TRD  and  Perf. 
The  percentage  of  projects  delivering  higher  performance  changed  from  13%  to  33%  to  52%  as 
SEC-TRD  increased  from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  43%  to 
33%  to  23%  as  SEC-TRD  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value 
of  +0.38  and  a  very  low  p-value  less  than  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  3 1  shows  the  relationship  between  SEC-TRD  and  Perf  for  those  projects  with  low¬ 
er  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  19  were  assessed  as  deploying  lower 
SEC-TRD  capabilities,  30  deploying  middle  SEC-TRD  capabilities,  and  24  deploying  higher 
SEC-TRD  capabilities. 

The  chart  shows  a  moderate  supporting  relationship  between  SEC-TRD  and  Perf  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  1 1%  to  33%  to  50%  as  SEC- 
TRD  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  26%  to  17% 
to  21%  as  SEC-TRD  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  increased  more  than  fourfold  with  improved  SEC-TRD,  while  that  of  de¬ 
livering  lower  performance  did  not  change  consistently.  This  relationship  is  characterized  by  a 
moderate  Gamma  value  of  +0.29  and  a  marginally  low  p-value  of  0.062. 
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Figure  31:  SEC-TRD  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  31  shows  the  relationship  between  5'£'C-J/?Z)  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  27  were  as¬ 
sessed  as  deploying  lowor  SEC-TRD  capabilities,  28  deploying  middle  SEC-TRD  capabilities, 
and  20  deploying  higher  SEC-TRD  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-TRD  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  15%  to  32%  to  55%  as  SEC- 
TRD  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  56%  to 
50%  to  25%  as  SEC-TRD  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  nearly  fourfold,  and  that  of  delivering  lower  performance 
was  reduced  to  less  than  half  with  improved  5'£'C-J/?Z).  This  relationship  is  characterized  by  a 
very  strong  Gamma  value  of  -1-0.43  and  a  very  low  p-value  of  0.004. 

We  infer  that  trade  studies  are  most  strongly  associated  with  better  performance  on  challenging 
projects,  which  may  depend  on  sound  trade  studies  to  evaluate  best  alternatives  in  support  of  deci¬ 
sion  making  for  difficult  or  complex  issues  and  to  reduce  potential  rework. 

5.4.6  Product  Integration 

The  distribution  of  the  responses  assessing  product  integration  activities  (SEC-PT)  of  the  projects 
is  shown  in  Figure  32  with  a  value  of  1  representing  very  poor  product  integration  and  4  repre¬ 
senting  very  good  product  integration. 
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1st  quartile  3.00 
Median  (2nd  quartile)  3.00 


Figure  32:  SEC-PI  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  product  integration  best  practices,  with 
room  for  improvement.  The  quartiles  are  all  listed  as  3.0  since  there  is  only  one  question  about 
product  integration,  and  over  half  of  the  survey  respondents  chose  the  third  option  (agree)  in  an¬ 
swering  the  question. 


Hence  we  could  not  use  the  usual  categorizing  criterion  in  this  one  instance  to  prepare  the  mosaic 
chart  showing  the  relationship  between  SEC-PI  and  Perf.  Instead  the  lower  group  includes  the 
projects  where  the  survey  respondents  chose  the  “strongly  disagree”  or  “disagree”  option.  The 
middle  group  includes  the  projects  where  the  respondents  chose  the  “agree”  category,  and  the 
higher  group  includes  the  projects  whose  respondents  chose  “strongly  agree”  in  answering  the 
question.  These  breakpoints  resulted  in  32  projects  categorized  as  having  lower  iS'£'C-P/ capabil¬ 
ity,  81  as  having  middle  5'£'C-P/ capability,  and  35  as  having  higher  iS'£'C-P/ capability.  The  re¬ 
sulting  mosaic  chart  is  shown  in  Figure  33. 
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Figure  33:  SEC-PI  vs.  Perf 
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Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  SEC-PI  and  Perf.  The 
percentage  of  projects  delivering  higher  performance  increased  from  16%  to  32%  to  49%  as  SEC- 
PI  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  50%  to  30% 
to  26%  as  SEC-PI  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value  of  +0.33 
and  a  very  low  p-value  of  0.003. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  34  shows  the  relationship  between  i'PC-P/ and  Pe// for  those  projects  with  lower 
PC  {PC  <  2.45).  This  set  contains  73  projects.  Of  these,  15  were  assessed  as  deploying  lower 
i'CC-P/ capabilities,  41  deploying  middle  i'CC-P/ capabilities,  and  17  deploying  higher  5'P'C-P/ 
capabilities. 

The  chart  shows  a  moderate  supporting  relationship  between  SEC-PI  and  Perf,  with  the  percent¬ 
age  of  projects  delivering  higher  performance  changing  from  20%  to  32%  to  47%  as  SEC-PI  in¬ 
creased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  33%  to  17% 
to  18%  as  SEC-PI  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  delivering 
higher  performance  more  than  doubled  with  improved  SEC-PI,  while  that  of  delivering  lower 
performance  was  reduced  by  nearly  half.  This  relationship  is  characterized  by  a  moderate  Gamma 
value  of  +0.23  and  a  marginal  p-value  of  0. 153. 


Figure  34:  SEC-VER  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  34  shows  the  relationship  between  SEC-PI  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.4).  This  set  contains  75  projects.  Of  these,  17  were  as¬ 
sessed  as  deploying  lower  SEC-PI  capabilities,  40  deploying  middle  SEC-PI  capabilities,  and  1 8 
deploying  higher  SEC-PI  capabilities. 
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The  chart  shows  a  very  strong  supporting  relationship  between  SEC-PI  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  12%  to  33%  to  50%  as  SEC-PI 
increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  65%  to 
43%  to  33%  as  SEC-PI  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of  de¬ 
livering  higher  performance  increased  fourfold,  and  that  of  delivering  lower  performance  was 
reduced  by  almost  half  with  improved  SEC-PI.  This  relationship  is  characterized  by  a  very  strong 
Gamma  value  of  +0.42  and  a  low  p-value  of  0.010. 

We  conclude  that  integration  best  practices  are  strongly  associated  with  the  performance  of  chal¬ 
lenging  projects  in  particular.  This  conclusion  seems  intuitive — ^that  stronger  integration  capabili¬ 
ties  are  necessary  to  achieve  the  best  performance  on  projects  that  are  large  or  complex.  Challeng¬ 
ing  projects  that  lack  a  strong  integration  capability  are  less  likely  to  achieve  top  levels  of 
performance. 

5.4.7  Verification 

The  distribution  of  the  responses  assessing  verification  activities  (SEC-VER)  of  the  projects  is 
shown  in  Figure  35  with  a  value  of  1  representing  very  poor  verification  work  and  4  representing 
very  good  verification  work. 

Istquartile  2.78 
Median  (2nd  quartile)  3.00 


Figure  35:  SEC-VER  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  verification  best  practices,  with  room  for 
improvement. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-VER  and  Perf,  three  groups 
for  SEC-VER  were  established  with  breakpoints  at  2.80  and  3.33.  These  breakpoints  resulted  in 
44  projects  categorized  as  having  lower  5'£'C-F£'/?  capability,  50  as  having  middle  SEC-VER 
capability,  and  54  as  having  hf^or  SEC-VER  capability.  The  resulting  mosaic  chart  is  shown  in 
Figure  36. 


CMU/SEI-2012-SR-009  |  42 


Higher 

Perf 


Middle 

Perf 


Lower 

Perf 


All 


Gamma  =  0.43  p-value  <  0.001 


Figure  36:  SEC-VER  vs.  Perf 


Examination  of  this  chart  reveals  a  very  strong  supporting  relationship  between  SEC-VER  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  from  16%  to  24%  to 
54%  as  SEC-VER  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  45%  to 
38%  to  19%  as  SEC-VER  increased.  This  relationship  is  characterized  by  a  very  strong  Gamma 
value  of  +0.43  and  a  very  low  p-value  less  than  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  37  shows  the  relationship  hQVNQQn  SEC-VER  and  Pej/ for  those  projects  with  low¬ 
er  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  23  were  assessed  as  deploying  lower 
SEC-VER  capabilities,  23  deploying  middle  SEC-VER  capabilities,  and  27  deploying  higher 
SEC-VER  capabilities. 

The  chart  shows  a  moderate  supporting  relationship  between  SEC-VER  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  22%  to  22%  to  52%  as  SEC- 
VER  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  22%  to  26% 
to  15%  as  SEC-VER  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  more  than  doubled  with  improved  SEC-VER,  while  that  of  delivering 
lower  performance  did  not  change  consistently.  This  relationship  is  characterized  by  a  moderate 
Gamma  value  of  +0.27  and  a  marginally  low  p-value  of  0.084. 


CMU/SEI-2012-SR-009  |  43 


Figure  37:  SEC-VER  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  37  shows  the  relationship  between  SEC-VER  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.4).  This  set  contains  75  projects.  Of  these,  21  were  as¬ 
sessed  as  deploying  lower  SEC-VER  capabilities,  27  deploying  middle  SEC-VER  capabilities, 
and  27  deploying  higher  iS'£'C-F£'/?  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-VER  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  10%  to  26%  to  56%  as  SEC- 
VER  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  71%  to 
48%  to  22%  as  SEC-VER  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  over  fivefold  and  that  of  delivering  lower  performance 
was  reduced  by  almost  70%  with  improved  SEC-VER.  This  relationship  is  characterized  by  a 
very  strong  Gamma  value  of  +0.60  and  a  very  low  p-value  less  than  0.001. 

We  conclude  that  verification  best  practices  are  strongly  associated  with  the  performance  of  chal¬ 
lenging  projects  in  particular.  This  conclusion  seems  intuitive — that  stronger  verification  capabili¬ 
ties  are  necessary  to  achieve  the  best  performance  on  projects  that  are  large  or  complex.  Challeng¬ 
ing  projects  that  lack  a  strong  verification  capability  rarely  achieve  top  levels  of  performance. 

5.4.8  Validation 

The  distribution  of  the  responses  assessing  validation  activities  (SEC-VAL)  of  the  projects  is 
shown  in  Figure  38  with  a  value  of  1  representing  very  poor  validation  work  and  4  representing 
very  good  validation  work. 
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Figure  38:  SEC-VAL  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  validation  best  practices,  with  room  for  im¬ 
provement. 


In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-VAL  and  Perf  three  groups 
for  SEC-VAL  were  established  with  breakpoints  at  2.70  and  3.30.  These  breakpoints  resulted  in 
36  projects  categorized  as  having  lower  SEC-VAL  capability,  73  as  having  middle  SEC-VAL 
capability,  and  39  as  having  hx^er  SEC-VAL  capability.  The  resulting  mosaic  chart  is  shown  in 
Figure  39. 
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Figure  39:  SEC-VAL  vs.  Perf 


Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  SEC-VAL  and  Perf. 
The  percentage  of  projects  delivering  higher  performance  changed  from  17%  to  27%  to  56%  as 
SEC-VAL  increased  from  lower  to  middle  to  higher 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  36%  to  38% 
to  21%  as  SEC-VAL  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value  of 
+0.33  and  a  very  low  p-value  of  0.003. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  40  shows  the  relationship  between  iSPC  VAL  and  Pej/ for  those  projects  with  lower 
PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  22  were  assessed  as  deploying  lower 
SEC-VAL  capabilities,  34  deploying  middle  SEC-VAL  capabilities,  and  17  deploying  higher 
SEC-VAL  capabilities. 


The  chart  shows  a  moderate  supporting  relationship  between  SEC-VAL  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  22%  to  24%  to  59%  as  SEC- 
VAL  increased  from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  18%  to  26% 
to  12%  as  SEC-VAL  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  more  than  doubled  with  improved  SEC-VAL,  while  that  of  delivering 
lower  performance  did  not  change  consistently.  This  relationship  is  characterized  by  a  moderate 
Gamma  value  of  +0.23  but  with  a  moderately  high  p-value  of  0. 127. 
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Figure  40:  SEC-VAL  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  40  shows  the  relationship  between  i’PC-FidZ,  and  Per/ for 
those  projects  with  higher  PC  {PC  >  2.4).  This  set  contains  75  projects.  Of  these,  14  were  as¬ 
sessed  as  deploying  lower  SEC-VAL  capabilities,  39  deploying  middle  SEC-VAL  capabilities, 
and  22  deploying  higher  SEC-VAL  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-VAL  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  0%  to  3 1%  to  55%  as  SEC- 
VAL  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  64%  to 
49%  to  27%  as  SEC-VAL  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  from  0%  to  over  50%,  and  that  of  delivering  lower  per¬ 
formance  was  reduced  by  almost  60%  with  improved  SEC-VAL.  This  relationship  is  character¬ 
ized  by  a  very  strong  Gamma  value  of  +0.48  and  a  very  low  p-value  of  0.002. 

We  conclude  that  validation  best  practices  are  strongly  associated  with  the  performance  of  chal¬ 
lenging  projects  in  particular.  This  conclusion  seems  intuitive — ^that  greater  involvement  with  end 
users  and  validation  of  project  performance  in  an  operational  environment  would  be  most  critical 
on  large  or  complex  projects,  and  such  projects  that  do  not  have  strong  validation  practices  are 
unlikely  to  achieve  the  best  levels  of  project  performance. 

5.4.9  Project  Monitoring  and  Controi 

The  distribution  of  the  responses  assessing  project  monitoring  and  control  activities  (SEC-PMC) 
of  the  projects  is  shown  in  Figure  41  with  a  value  of  1  representing  very  poor  project  monitoring 
and  control  and  4  representing  very  good  project  monitoring  and  control. 

Istquartile  2.78 
Median  (2nd  quartile)  3.00 


Figure  41:  SEC-PMC  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  project  monitoring  and  control  best  practic¬ 
es,  with  room  for  improvement. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-PMC  and  Petf,  three  groups 
for  SEC-TRD  were  established  with  breakpoints  at  2.90  and  3.30.  These  breakpoints  resulted  in 
48  projects  categorized  as  having  lovror  SEC-PMC  capability,  52  as  having  middle  SEC-PMC 
capability,  and  48  as  having  higher  SEC-PMC  capability.  The  resulting  mosaic  chart  is  shown  in 
Figure  42. 
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Figure  42:  SEC-PMC  vs.  Perf 


Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  SEC-PMC  and  Perf. 
The  percentage  of  projects  delivering  higher  performance  changed  from  17%  to  3 1%  to  50%  as 
SEC-PMC  increased  from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  46%  to  33% 
to  21%  as  SEC-PMC  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value  of 
+0.38  and  a  very  low  p-value  less  than  0.001. 


We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  43  shows  the  relationship  between  iSPC-PMC  and  Perf  for  those  projects  with  low¬ 
er  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  26  were  assessed  as  deploying  lower 
SEC-PMC  capabilities,  25  deploying  middle  SEC-PMC  capabilities,  and  22  deploying  higher 
SEC-PMC  capabilities. 

The  chart  shows  a  moderate  supporting  relationship  between  SEC-PMC  and  Perf  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  19%  to  36%  to  45%  as  SEC- 
PMC  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  27%  to  20% 
to  14%  as  SEC-PMC  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliv¬ 
ering  higher  performance  more  than  doubled  with  improved  SEC-PMC,  while  that  of  delivering 
lower  performance  was  reduced  by  nearly  half  This  relationship  is  characterized  by  a  moderate 
Gamma  value  of  +0.27  with  a  marginally  low  p-value  of  0.092. 
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Figure  43:  SEC-PMC  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  43  shows  the  relationship  between  SEC-PMC  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.4).  This  set  contains  75  projects.  Of  these,  22  were  as¬ 
sessed  as  deploying  lower  SEC-PMC  capabilities,  27  deploying  middle  SEC-PMC  capabilities, 
and  26  deploying  higher  SEC-PMC  capabilities. 

The  chart  shows  a  very  strong  supporting  relationship  between  SEC-PMC  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  14%  to  26%  to  54%  as  SEC- 
PMC  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  68%  to 
44%  to  27%  as  SEC-PMC  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  nearly  fourfold,  and  that  of  delivering  lower  performance 
was  reduced  by  more  than  half  with  improved  SEC-PMC.  This  relationship  is  characterized  by  a 
very  strong  Gamma  value  of  +0.53  and  a  very  low  p-value  less  than  0.001. 

We  conclude  that  a  strong  project  monitoring  and  control  capability  is  associated  with  higher  per¬ 
formance,  particularly  on  the  most  challenging  projects.  This  conclusion  seems  intuitive — ^that  the 
largest  and  most  complex  projects  depend  on  effective  project  monitoring  and  control  practices  to 
achieve  the  best  levels  of  project  performance. 

5.4.10  Risk  Management 

The  distribution  of  the  responses  assessing  risk  management  activities  (SEC-RSKM)  of  the  pro¬ 
jects  is  shown  in  Figure  44,  with  a  value  of  1  representing  very  poor  risk  management  and  4  rep¬ 
resenting  very  good  risk  management. 
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Figure  44:  SEC-RSKM  Response  Distribution 

The  median  of  3.00  indicates  moderate  application  of  risk  management  best  practices,  with  room 
for  improvement. 


In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-RSKM  and  Perf,  three 
groups  for  SEC-RSKM  were  established  with  breakpoints  at  2.80  and  3.36.  These  breakpoints 
resulted  in  50  projects  categorized  as  having  lower  capability,  45  as  having  middle 

SEC-RSKM  capability,  and  53  as  having  higher  capability.  The  resulting  mosaic 

chart  is  shown  in  Figure  45. 
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Figure  45:  SEC-RSKM  vs.  Perf 


Examination  of  this  chart  reveals  a  moderate  supporting  relationship  between  SEC-RSKM  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  from  24%  to  29%  to 
43%  as  SEC-RSKM  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  38%  to 
36%  to  26%  as  SEC-RSKM  increased.  This  relationship  is  characterized  by  a  moderate  Gamma 
value  of  +0.21  and  a  low  p-value  of  0.050. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  46  shows  the  relationship  between  SEC-RSKM  and  Perf  for  those  projects  with 
lower  PC  (PC  <  2.45).  This  set  contains  73  projects.  Of  these,  21  were  assessed  as  deploying  low¬ 
er  SEC-RSKM,  27  deploying  middle  SEC-RSKM,  and  25  deploying  higher  SEC-RSKM. 

The  chart  shows  a  weak  supporting  relationship  between  SEC-RSKM  and  Perf,  with  the  percent¬ 
age  of  projects  delivering  higher  performance  increasing  from  19%  to  33%  to  44%  as  SEC- 
RSKM  increased  from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  14%  to  33% 
to  12%  as  SEC-RSKM  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  de¬ 
livering  higher  performance  more  than  doubled  with  improved  SEC-RSKM,  but  that  of  delivering 
lower  performance  was  not  consistently  reduced.  The  lack  of  a  consistent  decrease  in  lower  per¬ 
forming  projects  as  SEC-RSKM  increased  may  reflect  the  influence  of  other  factors  impacting 
project  performance.  This  relationship  is  characterized  by  a  weak  Gamma  value  of  +0.18  and  a 
high  p-value  of  0.256. 
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Figure  46:  SEC-RSKM  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  46  shows  the  relationship  between  SEC-RSKM  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  29  were  as¬ 
sessed  as  deploying  lower  SEC-RSKM,  18  deploying  middle  SEC-RSKM,  and  28  deploying 
higher 


The  chart  shows  a  moderate  supporting  relationship  between  SEC-RSKM  and  Petf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  28%  to  22%  to  43%  as  SEC- 
RSKM  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  55%  to  39% 
to  39%  as  SEC-RSKM  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of  de¬ 
livering  higher  performance  increased  by  over  50%,  and  that  of  delivering  lower  performance  was 
reduced  by  nearly  30%  with  improved  SEC-RSKM.  This  relationship  is  characterized  by  a  mod¬ 
erate  Gamma  value  of  +0.24  and  a  moderately  high  p-value  of  0. 124. 

Overall,  higher  SEC-RSKM  capability  appears  associated  with  better  project  performance,  but  to 
a  lesser  degree  than  most  other  factors.  This  association  may  be  due  in  part  to  a  response  distribu¬ 
tion  that  is  more  heavily  weighted  toward  higher  risk  management  capability  scores,  which  may 
suggest  generally  mature  risk  management  processes  with  less  variation  among  higher  and  lower 
performing  projects. 

5.4.11  Configuration  Management 

The  distribution  of  the  responses  assessing  configuration  management  activities  (SEC-CM)  of  the 
projects  is  shown  in  Figure  47  with  a  value  of  1  representing  very  poor  configuration  management 
and  4  representing  very  good  configuration  management. 

Istquartile  3.00 
Median  (2nd  quartile)  3.40 


Figure  47:  SEC-CM  Response  Distribution 

The  median  of  3.40  indicates  strong  application  of  configuration  management  best  practices,  with 
a  little  room  for  improvement. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-CM  and  Perf,  three  groups 
for  SEC-TRD  were  established  with  breakpoints  at  3.10  and  3.80.  These  breakpoints  resulted  in 
59  projects  categorized  as  having  lower  5'£'C-CM  capability,  51  as  having  middle  SEC-CM  capa¬ 
bility,  and  38  as  having  higher  SEC-CM  capability.  The  resulting  mosaic  chart  is  shown  in  Figure 
48. 
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Figure  48:  SEC-CM  vs.  Perf 


Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  SEC-CM  and  Perf. 

The  percentage  of  projects  delivering  higher  performance  changed  from  17%  to  39%  to  47%  as 
SEC-CM  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  46%  to  27% 
to  21%  as  SEC-CM  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value  of 
+0.38  and  a  very  low  p-value  of  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  49  shows  the  relationship  between  SEC-CM  and  Perf  for  those  projects  with  lower 
PC  {PC  <  2.45).  This  set  contains  73  projects.  Of  these,  15  were  assessed  as  deploying  lower 
SEC-CM  capabilities,  41  deploying  middle  5'£'C-CM  capabilities,  and  17  deploying  higher  5'£'C- 
CM  capabilities. 

The  chart  shows  a  moderate  supporting  relationship  between  SEC-CM  and  Perf  with  the  per¬ 
centage  of  projects  delivering  higher  performance  changing  from  20%  to  44%  to  39%  as  SEC- 
CM  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  27%  to  16% 
to  17%  as  SEC-CM  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  nearly  doubled  with  improved  SEC-CM,  while  that  of  delivering  lower 
performance  was  reduced  by  more  than  one-third.  This  relationship  is  characterized  by  a  moderate 
Gamma  value  of  +0.22  but  with  a  moderately  high  p-value  of  0.203. 
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Figure  49:  SEC-CM  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  49  shows  the  relationship  between  5'£'C-CM  and  Perf  for 
those  projects  with  higher  PC  {PC  >  2.4).  This  set  contains  75  projects.  Of  these,  29  were  as¬ 
sessed  as  deploying  lower  5'£'C-CM  capabilities,  26  deploying  middle  SEC-CM  capabilities,  and 
20  deploying  higher  SEC-CM  capabilities. 


The  chart  shows  a  very  strong  supporting  relationship  between  SEC-CM  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  14%  to  35%  to  55%  as  SEC- 
CM  increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  65%  to 
38%  to  25%  as  SEC-CM  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of 
delivering  higher  performance  increased  nearly  fourfold,  and  that  of  delivering  lower  performance 
was  reduced  by  more  than  half  with  improved  SEC-CM.  This  relationship  is  characterized  by  a 
very  strong  Gamma  value  of  -1-0.53  and  a  very  low  p-value  less  than  0.001. 


We  conclude  that  a  strong  configuration  management  capability  is  strongly  associated  with  higher 
performance  on  the  most  challenging  projects.  This  conclusion  seems  intuitive — ^that  the  largest 
and  most  complex  projects  depend  most  heavily  on  best  practices  for  configuration  management, 
and  such  projects  lacking  effective  configuration  management  activities  are  unlikely  to  achieve 
the  best  levels  of  project  performance. 


5.4.12  Integrated  Product  Teams 


The  distribution  of  the  responses  assessing  the  use  of  integrated  product  teams  (SEC-IPT)  is 
shown  in  Figure  50,  with  a  value  of  1  representing  very  poor  use  and  4  representing  very  good 
use. 
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Figure  50:  SEC-IPT  Response  Distribution 

The  median  of  3.00  indicates  moderate  use  of  IPTs,  with  room  for  improvement. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  SEC-IPT  and  Perf,  three  groups 
for  SEC-IPT  were  established  with  breakpoints  at  2.60  and  3.00.  These  breakpoints  resulted  in  51 
projects  categorized  as  having  lowsr  SEC-IPT  capability,  52  as  having  middle  SEC-IPT  capabil¬ 
ity,  and  45  as  having  higher  SEC-IPT  capability.  The  resulting  mosaic  chart  is  shown  in  Figure 
51. 
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Figure  51:  SEC-IPT  vs.  Perf 


Examination  of  this  chart  reveals  only  a  weak  supporting  relationship  between  SEC-IPT  and 
Perf.  The  percentage  of  projects  delivering  higher  performance  increased  only  from  22%  to  35% 
to  42%  as  SEC-IPT  increased  from  lower  to  middle  to  higher.  Similarly,  the  percentage  of  pro¬ 
jects  delivering  lower  performance  changed  from  37%  to  31%  to  31%  as  SEC-IPT  increased  from 
lower  to  middle  to  higher.  This  relationship  is  characterized  by  a  weak  Gamma  value  of  +0. 18 
with  a  moderately  high  p-value  of  0. 1 0 1 . 
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We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  52  shows  the  relationship  between  5£'C-/Pr  and  Pe;/ for  those  projects  with  lower 
PC  {PC  <  2.45).  This  set  contains  73  projects.  Of  these,  23  were  assessed  as  deploying  lower 
SEC-IPT  capabilities,  26  deploying  middle  SEC-IPT  capabilities,  and  24  deploying  higher  iSPC- 
IPT  capabilities. 

The  chart  shows  a  weak  opposing  relationship  between  SEC-IPT  and  Perf,  with  the  percentage  of 
projects  delivering  higher  performance  changing  from  30%  to  38%  to  29%  as  SEC-IPT  increased 
from  lower  to  middle  to  higher. 


Additionally,  the  percentage  of  projects  delivering  lower  performance  increased  from  13%  to  23% 
to  25%  as  SEC-IPT  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  did  not  change  appreciably  with  improved  SEC-IPT,  while  those  deliver¬ 
ing  lower  performance  almost  doubled.  This  relationship  is  characterized  by  a  weak  Gamma  value 
of  +0. 12  and  a  high  p-value  of  0.463,  indicating  it  is  difficult  to  draw  any  reliable  conclusions 
from  this  sample.  SEC-IPT  does  not  appear  to  have  a  significant  correlation  with  the  performance 
of  less  challenging  projects. 
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Figure  52:  SEC-IPT  vs.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  52  shows  the  relationship  between  iS'£'C-/PJ  and  Per/ for 
those  projects  with  higher  PC  {PC  >  2.45).  This  set  contains  75  projects.  Of  these,  28  were  as¬ 
sessed  as  deploying  lower  SEC-IPT  capabilities,  26  deploying  middle  SEC-IPT  capabilities,  and 
21  deploying  higher  ^PC-ZPr  capabilities. 


The  chart  shows  a  very  strong  supporting  relationship  between  SEC-IPT  and  Perf,  with  the  per¬ 
centage  of  projects  delivering  higher  performance  increasing  from  14%  to  31%  to  57%  as  SEC- 
IPT  increased  from  lower  to  middle  to  higher. 
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Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  57%  to  38% 
to  38%  as  SEC-IPT  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of  deliver¬ 
ing  higher  performance  increased  more  than  fourfold  and  that  of  delivering  lower  performance 
decreased  by  one-third  with  improved  SEC-IPT.  This  relationship  is  characterized  by  a  very 
strong  Gamma  value  of  +0.40  and  a  low  p-value  of  0.007. 

These  findings  suggest  higher  SEC-IPT  capability  has  a  greater  effect  on  more  challenging  pro¬ 
jects,  which  might  be  expected  to  depend  more  heavily  on  integrated  cross-functional  involve¬ 
ment  to  successfully  solve  the  most  complex  problems,  interfaces,  or  larger  projects. 

5.5  Other  Factors 

The  following  five  sections  discuss  the  results  of  the  survey  for  these  other  factors:  experience, 
contract  type,  SE  organization,  project  percentage  complete,  and  SE  content. 

5.5.1  Experience 

The  distribution  of  the  responses  assessing  prior  experience  {EXP)  with  similar  projects  is  shown 
in  Figure  53  with  a  value  of  1  representing  very  little  prior  experience  and  4  representing  a  lot  of 
prior  experience. 

Istquartile  2.50 
Median  (2nd  quartile)  3.00 


Figure  53:  EXP  Response  Distribution 

The  median  of  3.00  indicates  that  the  sampled  projects  were  just  above  the  middle  of  the  continu¬ 
um  of  prior  experience. 

In  preparing  the  mosaic  chart  showing  the  relationship  between  EXP  and  Perf,  three  groups  for 
EXP  were  established  with  breakpoints  at  2.60  and  3.40.  These  breakpoints  resulted  in  44  projects 
categorized  as  having  lower  EXP,  67  as  having  middle  EXP,  and  37  as  having  higher  EXP.  The 
resulting  mosaic  chart  is  shown  in  Figure  54. 
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Figure  54:  EXP  vs.  Perf 


Examination  of  this  chart  reveals  a  strong  supporting  relationship  between  EXP  and  Perf.  The 
percentage  of  projects  delivering  higher  performance  changed  from  27%  to  24%  to  54%  as  EXP 
increased  from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  50%  to 
3 1%  to  16%  as  EXP  increased.  This  relationship  is  characterized  by  a  strong  Gamma  value  of 
+0.36  and  a  very  low  p-value  of  0.001. 

We  further  examined  this  relationship  by  incorporating  the  impact  of  PC.  The  chart  on  the  left 
side  of  Figure  55  shows  the  relationship  between  EXP  and  Perf  for  those  projects  with  lower  PC 
{PC  <  2.45).  This  set  contains  73  projects.  Of  these,  18  were  assessed  as  deploying  lower  EAP,  27 
deploying  middle  EXP,  and  28  deploying  higher  £WP. 

The  chart  shows  a  very  strong  supporting  relationship  between  EXP  and  Perf  with  the  percentage 
of  projects  delivering  higher  performance  changing  from  19%  to  36%  to  45%  as  EXP  increased 
from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  changed  from  27%  to  20% 
to  14%  as  EXP  increased.  Thus,  for  these  lower  challenge  projects,  the  likelihood  of  delivering 
higher  performance  more  than  doubled  with  improved  EXP,  while  that  of  delivering  lower  per¬ 
formance  was  reduced  by  nearly  half.  This  relationship  is  characterized  by  a  very  strong  Gamma 
value  of  +0.51  and  a  very  low  p-value  of  0.001. 
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Figure  55:  EXP  i/s.  Perf  Controlled  by  PC 


The  chart  on  the  right  side  of  Figure  55  shows  the  relationship  between  EXP  and  Perf  for  those 
projects  with  higher  PC  {PC>  2.4).  This  set  contains  75  projects.  Of  these,  26  were  assessed  as 
deploying  lower  EXP,  40  deploying  middle  EXP,  and  9  deploying  higher  EXP. 

The  chart  shows  a  weak  supporting  relationship  between  EXP  and  Perf,  with  the  percentage  of 
projects  delivering  higher  performance  increasing  from  3 1%  to  30%  to  44%  as  EXP  increased 
from  lower  to  middle  to  higher. 

Additionally,  the  percentage  of  projects  delivering  lower  performance  decreased  from  58%  to 
40%  to  33%  as  EXP  increased.  Thus,  for  these  higher  challenge  projects,  the  likelihood  of  deliv¬ 
ering  higher  performance  increased  by  more  than  one-third,  and  that  of  delivering  lower  perfor¬ 
mance  was  reduced  by  more  than  one-third  with  improved  EXP.  This  relationship  is  characterized 
by  a  weak  Gamma  value  of  +0.19  and  a  high  p-value  of  0.258. 

These  findings  seem  consistent  with  our  intuition.  Projects  are  considered  challenging  or  not 
largely  based  on  our  past  experience  with  similar  applications.  There  is  a  strong  direct  relationship 
between  this  past  experience  and  the  likely  success  of  future  projects.  Projects  that  include  dealing 
with  complexity,  incorporating  innovation,  or  pushing  the  boundaries  of  “state  of  the  art”  while 
having  little  prior  experience,  are  challenging  projects.  When  we  have  little  relevant  experience  to 
fall  back  on,  project  success  is  likely  to  depend  on  other  critical  factors  such  as  the  use  of  strong 
systems  engineering  practices. 

5.5.2  Contract  Type 

The  survey  explored  the  type  of  contract  governing  the  projects.  Respondents  were  asked  to 
choose  between  the  following  contract  types: 

fixed-price — The  total  contract  value  is  primarily  determined  by  the  initial  contract  (e.g.,  firm 
fixed  price  [FFP],  fixed  price  incentive  fee  [FPIF],  firm  fixed  price-level  of  effort  [FFP-LOE]). 
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cost-reimbursable — The  total  contract  value  is  primarily  determined  by  the  buyer’s  cost  of  exe¬ 
cuting  the  contract  (e.g.,  cost  plus  fixed  fee  [CPFF],  cost  plus  award  fee  [CPAF],  cost  plus  incen¬ 
tive  fee  [CPIF]). 

other  — The  contract  is  a  type  that  does  not  fit  the  prior  two  categories. 

If  they  chose  “other  contract  type,”  respondents  were  asked  to  describe  the  contract  type.  Typical 
responses  included 

•  a  mix  of  FFP  and  CPFF 

•  task-order  contract 

The  distribution  of  responses  assessing  contract  type  is  shown  in  Figure  56. 


What  type  of  contract(s)  was  awarded  for  this  project? 
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Figure  56:  Contract  Type  Response  Distribution 


The  mosaic  chart  showing  the  performance  distribution  for  each  of  these  contract  types  is  shown 
in  Figure  57. 
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Figure  57:  Contract  Type  vs.  Perf 
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This  chart  shows  that  cost-reimbursable  contracts  perform  slightly  worse  than  fixed  price  con¬ 
tracts,  with  the  percentage  of  projects  delivering  higher  performance  dropping  from  36%  to  32%, 
and  the  percentage  of  projects  delivering  lower  performance  increasing  from  28%  to  36%.  This 
result  is  consistent  with  intuition,  in  that  cost-reimbursable  contracts  are  typically  used  for  riskier 
projects;  hence  lower  project  performance  is  not  unexpected. 

Performance  differences  between  contract  types  do  not  appear  to  be  significant.  Additional  tests 
of  data  relationships  factoring  in  the  impact  of  project  challenge  did  not  add  additional  insight. 

5.5.3  SE  Organization 

The  survey  explored  the  impact  of  the  structure  of  the  SE  organization  within  the  company  exe¬ 
cuting  the  project.  SE  may  be  either  centralized  in  an  SE  department  or  distributed  throughout 
other  departments  within  the  company.  Respondents  were  asked  to  choose  between  the  following 
statements: 

•  Systems  engineering  skills  and  responsibilities  are  contained  in  a  separate  department. 

•  Systems  engineering  skills  and  responsibilities  are  distributed  throughout  other  depart¬ 
ments. 

The  distribution  of  the  responses  assessing  the  SE  organization  is  shown  in  Figure  58. 


Responses  were  nearly  evenly  divided  between  the  two  alternatives. 

We  first  explored  the  relationship  between  SE-Orgn  and  the  deployment  of  SE  best  practices,  as 
measured  hy  S EC-Total.  We  divided  the  responding  projects  into  those  with  centralized  SE  (85 
projects)  and  those  with  distributed  SE  (63  projects).  We  then  examined  the  distribution  of  SE 
deployment,  as  measured  by  SEC-Total,  in  each  of  these  groups.  In  breaking  SEC-Total  into 
three  groups,  we  used  the  previously  established  breakpoints  of  2.85  and  3.27  as  discussed  in  Sec¬ 
tion  5.4.1.  The  resulting  mosaic  chart  is  shown  in  Figure  59. 
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Figure  59:  SE-Orgn  vs.  SEC-Total 


This  chart  shows  that  organizations  with  centralized  SE  are  somewhat  more  effective  at  deploying 
SE  best  practices  when  compared  to  organizations  with  distributed  SE.  The  obvious  question  to 
ask  is,  “Does  this  more  effective  deployment  of  SE  best  practices  influence  project  performance?” 
To  explore  this  question,  we  examined  the  relationship  between  SE-Orgn  and  Perf  via  the  mosaic 
chart  of  Figure  60. 
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Figure  60:  SE-Orgn  vs.  Perf 


We  see  virtually  no  difference  in  performance  between  organizations  with  centralized  SE  versus 
those  with  distributed  SE.  Flow  do  we  reconcile  this  result  with  the  fact  that  Figure  19  shows  that 
better  deployment  of  SE  is  related  to  better  performance,  and  Figure  60  shows  that  organizations 
with  centralized  SE  provide  somewhat  better  deployment  of  SE?  One  hypothesis  is  that  while 
centralized  SE  is  more  effective  at  performing  SE  tasks  and  producing  the  SE  artifacts  assessed  in 
this  survey,  those  efforts  and  artifacts  are  less  integrated  with  the  work  of  the  project  and  therefore 
have  less  impact  on  project  performance.  This  hypothesis  is  worthy  of  additional  study. 
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5.5.4  Project  Percentage  Complete 


Nearly  all  projects  start  out  on  budget  and  on  schedule.  It  is  only  as  project  execution  progresses 
that  departures  from  the  plan  occur.  Thus,  you  might  expect  that  projects  in  the  early  phases  of 
execution  would  predict  performance  at  completion  to  be  very  close  to  the  initial  plan,  while  pro¬ 
jects  in  the  later  phases  of  execution  would  be  more  likely  to  recognize  and  predict  deviations. 

The  survey  tested  this  hypothesis  by  capturing  data  regarding  the  percentage  complete  of  the  pro¬ 
ject.  Distribution  of  these  responses  is  shown  in  Figure  61. 


What  is  the  current  completion  status  of  this  project? 


70 


Figure  61:  SE  Project  Percentage  Complete  Response  Distribution 

Half  of  the  projects  responding  to  the  survey  were  71%  complete  (Figure  60).  This  is  a  sufficient 
degree  of  completion  to  reflect  the  production  of  the  SE  artifacts  queried  in  the  survey  and  also  a 
sufficient  degree  of  completion  to  develop  some  confidence  in  performance  projections. 

We  used  the  scatter  plot  in  Figure  62  to  examine  the  relationship  between  percentage  complete 
and  Perf. 
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The  solid  line  represents  a  linear  fit  of  the  data.  Examination  of  this  chart  does  not  support  the 
hypothesis  postulated  earlier  in  this  section  and  shows  that  neither  project  performance  nor  the 
variance  in  project  performance  shows  much  relationship  to  the  percentage  of  project  completion. 

5.5.5  SE  Content 

The  survey  captured  data  regarding  the  amount  of  SE  effort  contained  in  the  project  with  the 
question. 

Approximately  what  percentage  of  non-recurring  engineering  (NRE)  does  Systems  Engineer¬ 
ing  represent? 

Distribution  of  these  responses  is  shown  in  Figure  63. 
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Figure  63:  %SE  Response  Distribution 

The  mean  and  median  values  for  all  projects  were  22%  and  15%  respectively.  Projects  with  very 
high  percentages  of  SE  effort  may  indicate  projects  that  are  not  traditional  development  projects 
(e.g.,  study  contracts).  We  examine  the  relationship  between  %SE  and  Perf  using  the  scatter  plot 
in  Figure  64. 
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Figure  64:  %SE  vs.  Perf 


The  solid  line  represents  a  linear  fit  of  the  data.  Examination  of  these  data  does  not  show  a  signif¬ 
icant  relationship  between  project  performance  and  the  amount  of  SE  applied  to  the  project  as  a 
percentage  of  total  NRE. 
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6  Comparison  with  the  Prior  Study 


This  study  is  similar  to  one  conducted  in  2007  [Elm  2008].  However,  this  study  was  conducted 
with  some  differences  from  the  first  and  the  results  also  varied  from  the  first. 

6.1  Comparison  of  Study  Executions 

The  following  sections  describe  the  differences  between  the  questionnaire,  analysis,  and  sampling 
used  in  the  prior  study  to  those  used  in  this  study. 

6.1.1  Questionnaire  Differences 

While  the  tools  and  methods  used  in  this  study  are  very  similar  to  those  used  in  the  2007  NDIA 
study  [Elm  2008],  some  changes  were  made  based  on  lessons  learned  from  that  study.  Primary 
complaints  about  the  prior  study  centered  on  the  length  of  the  questionnaire  and  the  difficulties  in 
answering  some  of  the  questions.  In  response  to  these  complaints,  we  modified  the  questionnaire 
in  the  following  areas: 

1 .  Elimination  of  questions  addressing  aequirer  eapabilities  -  In  the  2007  study,  we  hy¬ 
pothesized  the  capabilities  of  the  acquirer  were  a  factor  affecting  the  overall  performance 
of  the  project.  To  test  this  hypothesis,  the  2007  study  included  five  questions  intended  to 
assess  acquirer  capabilities.  Analysis  of  the  responses  showed  a  moderately  strong  nega¬ 
tive  relationship  between  the  acquirer’s  capabilities  and  project  performance — a  puzzling 
result.  While  we  believe  that  this  result  requires  further  study,  we  also  determined  that 
such  investigation  would  require  an  expansion  of  the  questionnaire  to  probe  more  deeply 
into  acquirer  capabilities.  Since  this  was  not  the  primary  focus  of  this  study  (i.e.,  as¬ 
sessing  the  effectiveness  of  SE  practices),  we  chose  to  eliminate  this  area  of  study. 

2.  Simplification  of  the  process  of  assessing  project  challenge  -  In  both  studies,  project 
challenge  {PC)  was  assessed  using  the  factors  in  Table  7.  Notice  that  the  2007  study  used 
more  factors  than  the  2012  study. 
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Table  7:  Factors  Used  to  Assess  Project  Challenge 


PC  Assessment  Factor 

2007 

2012 

Included  lifecycle  phases 

• 

Sources  of  technical  challenge 

• 

• 

Inter-organizational  complexity 

• 

Contract  duration 

• 

• 

Contract  stability 

• 

Change  in  contract  duration 

• 

Lifecycle  phases  currently  in  execution 

• 

Total  project  effort 

• 

• 

Contract  value 

• 

• 

Requirements  completeness  and  stability 

• 

• 

Percentage  change  of  contract  value 

• 

• 

Dollar  change  of  contract  value 

• 

Percentage  change  in  contract  duration 

• 

3.  Reduction  of  the  assessment  of  project  environment  factors  The  2007  study  investi¬ 
gated  a  number  of  environmental  factors  that  were  hypothesized  to  have  an  impact  on 
project  performance.  These  factors  included 

•  customer  type  •  acquiring  organization 

•  end  user  •  contract  type 

•  system  deployment  environment  •  CMMI-related  capabilities 

•  percent  of  effort  subcontracted  •  prior  experience 

•  process  improvement  efforts  •  position  in  the  systems  hierarchy  (e.g., 

SoS  [system  of  systems],  system,  sub¬ 
system) 

Results  from  the  analysis  of  some  of  these  data  were  not  particularly  useful.  Furthermore, 
since  this  study  was  targeting  a  broader  and  more  diverse  population  (global  system  de¬ 
velopers  from  all  industries  rather  than  just  U.S.  defense  contractors),  some  of  the  factors 
were  no  longer  relevant.  To  simplify  the  questionnaire,  the  environmental  factors  as¬ 
sessed  were  reduced  to 

•  customer  type  •  contract  type 

•  end  user  •  prior  experience 

4.  Simplification  of  the  assessment  process  for  project  monitoring  and  control  -  The 

2007  study  included  1 1  questions  assessing  SEC-PMC.  These  were  reduced  to  eight 
questions,  eliminating  the  inquiries  about  operational  hazard  risk  assessments,  materiel 
readiness  assessments,  and  system  upgrade  planning. 

5.  Improvement  of  the  assessment  process  for  risk  management  -  The  2007  study  in¬ 
cluded  five  questions  assessing  SEC-RSKM.  This  study  added  three  more,  assessing  the 
integration  of  risk  management  with  cost  and  schedule  management  and  with  supplier 
risk  management. 
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6.  Reduction  of  the  assessment  process  for  product  architecture  -  The  2007  study  in¬ 
cluded  six  questions  assessing  SEC-ARCH.  These  were  reduced  to  five  questions,  elimi¬ 
nating  the  inquiry  about  the  management  of  commercial  off-the-shelf  (COTS)  products. 

6.1.2  Analysis  Differences 

In  addition  to  the  modifications  to  the  questionnaire,  we  also  made  improvements  to  the  analysis 
process — ^particularly  the  analysis  of  project  performance.  The  2007  study  assessed  project  per¬ 
formance  by  examining  factors  addressing  cost  performance  (five  questions),  schedule  perfor¬ 
mance  (seven  questions),  and  technical  performance  (one  question).  These  factors  were  then  com¬ 
bined  into  a  weighted  summed  index  to  create  an  overall  performance  assessment.*^  This  process 
gave  undue  weight  to  the  single  question  assessing  technical  performance,  making  it  as  significant 
as  all  of  the  questions  assessing  schedule  or  cost  performance  taken  together.  In  this  study,  we 
improved  the  assessment  process  for  performance  by  using  ten  questions  to  assess  cost  perfor¬ 
mance,  eight  questions  to  assess  schedule  performance,  and  two  questions  to  assess  technical  per¬ 
formance. 

Furthermore,  we  increased  the  rigor  applied  to  the  performance  assessment  process.  In  the  2007 
study,  some  human  interpretation  of  the  survey  responses  was  needed  to  develop  the  performance 
assessment.  This  interpretation  was  done  by  the  researchers  and  validated  by  independent  experts 
in  project  management.  In  this  study,  we  were  able  to  improve  the  questionnaire  and  codify  the 
analysis  process  in  a  manner  that  used  no  additional  human  judgment.  The  analysis  process  is 
documented  in  the  report  The  Business  Case  for  Systems  Engineering  Study:  Assessing  Project 
Performance  from  Sparse  Data  [Elm  2012].  The  process  was  reviewed  and  validated  by  inde¬ 
pendent  experts  in  project  management. 

6.1.3  Sampling  Differences 

In  the  2007  study,  participants  were  solicited  through  the  NDIA-SED  membership.  Obviously, 
this  method  limited  the  sample  to  members  of  the  U.S.  defense  industrial  sector.  For  this  study, 
we  reached  out  not  only  to  the  NDIA  members,  but  also  to  the  members  of  the  lEEE-AESS  and 
INCOSE.  Both  of  these  organizations  are  international  and  cover  a  multitude  of  industrial  sectors 
beyond  U.S.  defense. 

We  also  chose  multiple  methods  of  soliciting  respondents.  In  addition  to  broadly  soliciting  the 
memberships  of  the  NDIA,  IEEE,  and  INCOSE,  we  also  targeted  a  number  of  organizations  rep¬ 
resented  within  these  groups.  Within  each  of  the  targeted  organizations,  we  identified  a  sponsor 
who  would  recruit  respondents  and  manage  their  participation  in  the  survey.  The  organizations 
targeted  were  typically  those  that  were  familiar  with  the  2007  study  and  had  found  it  useful.  Since 
the  2007  study  was  centered  on  U.S.  defense  contractors  and  published  through  the  NDIA  and 
SEI,  most  of  the  organizations  familiar  with  it  were  U.S.  defense  contractors;  thus  most  of  the 
organizations  targeted  for  this  study  were  also  U.S.  defense  contractors.  As  shown  in  Figures  6,  7, 
and  8,  the  majority  of  the  responses  come  from  U.S.  defense  contractors. 


A  description  of  the  calculation  of  weighted  summed  indices  can  be  found  in  Section  4.1  on  page  10. 
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6.2  Differences  in  Study  Resuits 

The  results  of  this  study  reinforce  those  of  the  2007  study;  however,  some  differences  are  appar¬ 
ent.  Figure  65  shows  the  mosaic  chart  from  the  2007  study  detailing  the  relationship  between  pro¬ 
ject  performance  and  the  total  deployment  of  SE  as  assessed  by  the  questionnaire. 
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p  =  0.04 


Figure  65:  Perfvs.  SEC-Total  (from  2007  NDIA  Study) 


The  chart  in  Figure  65  from  the  2007  study  is  remarkably  similar  to  the  chart  produced  by  this 
study,  shown  in  Figure  66.  Both  studies  showed  an  increase  in  projects  delivering  higher  perfor¬ 
mance,  and  a  decrease  in  projects  delivering  lower  performance  as  SE  deployment  increases. 
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Figure  66:  Perf  vs.  SEC-Total 


In  the  2007  research  study,  we  also  examined  the  relationships  between  project  performance  and 
specific  SE  process  groups,  just  as  we  did  in  this  study.  For  each  process  group.  Table  8  shows  the 
Gamma  value  and  p-value  of  its  relationship  to  project  performance  for  both  the  2007  and  the 
2012  studies. 


Table  8:  Comparison  of  Gamma  Values  and  p-values 


SE  Process  Area 

Gamma 

p-value 

2007 

2012 

2007 

2012 

SEC-Total 

+  0.32 

+  0.49 

0.040 

<0.001 

SEC-PP 

+  0.13 

+  0.46 

0.250 

<0.001 

SEC-REQ 

+  0.33 

+  0.44 

0.040 

<0.001 

SEC-VER 

+  0.25 

+  0.43 

0.090 

<0.001 

SEC-ARCH 

+  0.40 

+  0.41 

0.002 

<0.001 

SEC-CM 

+  0.13 

+  0.38 

0.260 

0.001 

SEC-TRD 

+  0.37 

+  0.38 

0.030 

<0.001 

SEC-PMC 

-0.13 

+  0.38 

0.250 

<0.001 

SEC-VAL 

+  0.28 

+  0.33 

0.070 

0.003 

SEC-PI 

+  0.21 

+  0.33 

0.160 

0.003 

SEC-RSKM 

+  0.28 

+  0.21 

0.061 

0.050 

SEC-IPT 

+  0.34 

+  0.18 

0.040 

0.101 
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A  summary  of  this  comparison  is  shown  in  Figure  67. 


Most  results  from  the  two  studies  are  generally  in  agreement,  which  is  reasonable  for  data  of  this 
nature.  Notable  differences  in  the  results  of  the  two  studies  exist  for  the  relationships  between 
project  performance  (Perf)  and  four  systems  engineering  capabilities. 

1.  SEC-PP—  In  the  2007  study,  this  relationship  was  evaluated  at  Gamma  =  +  0.13.  Now  it  is 
evaluated  at  Gamma  =  +  0.46,  which  is  a  considerable  increase  from  what  was  a  weak  rela¬ 
tionship  to  a  very  strong  one. 

As  noted  in  Section  6. 1 . 1 ,  the  questions  assessing  SEC-PP  did  not  change  appreciably  be¬ 
tween  the  two  studies.  As  noted  in  Section  6.1.2,  improvements  were  made  in  the  questions 
and  process  for  assessing  Perf. 

It  is  conceivable  that  these  changes  account  for  the  observed  differences  in  results.  To  assess 
this  possibility,  we  reverted  to  an  analysis  process  more  similar  to  the  one  used  in  2007. 
While  it  was  not  practical  to  use  the  less  rigorous  assessment  process  of  the  2007  study,  we 
could  eliminate  some  of  the  Perf  questions  added  for  this  study.  Once  we  eliminated  those 
questions,  we  found  that  47  of  the  148  responses  used  in  the  analysis  were  rendered  unusable 
due  to  insufficient  data  to  evaluate  Perf  reducing  the  number  of  useable  cases  to  101. 

Examining  the  relationship  between  SEC-PP  and  Perf  for  these  101  cases  did  not  produce  a 
result  more  consistent  with  the  2007  study.  Hence,  it  is  reasonable  to  claim  that  the  changes 
in  the  assessment  of  Perf  do  not  account  for  the  observed  relationship  difference.  However 
the  explanation  may  lie  in  differences  between  the  two  survey  samples. 
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2.  SEC-CM  —  In  the  2007  study,  this  relationship  was  evaluated  at  Gamma  =  +0.13.  Now,  it  is 
evaluated  at  Gamma  =  +0.38,  which  also  is  a  considerable  increase  from  what  was  a  weak 
relationship  to  a  strong  one. 

Similar  to  SEC-PP,  the  questions  assessing  SEC-CM  did  not  change  appreciably  between 
the  two  studies;  however  the  questions  and  process  for  assessing  Perf  did.  To  assess  the  im¬ 
pact  of  this  change,  we  repeated  the  analysis  eliminating  the  additional  Perf  questions  and 
found  that  the  analysis  of  the  relationship  between  SEC-PP  and  Perf  did  not  produce  a  result 
more  consistent  with  the  2007  study. 

Hence,  it  is  reasonable  to  claim  that  the  changes  in  the  assessment  of  Perf  do  not  account  for 
the  observed  relationship  difference.  However  the  explanation  may  lie  in  differences  be¬ 
tween  the  two  survey  samples. 

3.  SEC-PMC  —  In  the  2007  study,  this  relationship  was  evaluated  at  Gamma  =  -0. 13.  Now,  it  is 
evaluated  at  Gamma  =  +0.38.  That  is,  of  course,  a  very  large  difference  in  magnitude  that  al¬ 
so  is  a  change  from  a  weak  negative  relationship  to  a  strong  positive  one. 

As  noted  in  Section  6.1.1,  this  study  deleted  several  questions  that  were  used  to  assess  SEC- 
PMC  in  the  2007  study.  These  questions  centered  on  the  use  of  engineering  analyses  to  sup¬ 
port  operational  hazard  risk  assessments,  materiel  readiness  assessments,  and  system  up¬ 
grade  planning.  These  topics  tend  to  be  specialty  subjects  “beyond  the  mainstream”  funda¬ 
mentals  of  project  monitoring  and  control  that  may  not  apply  to  the  set  of  sampled  projects  at 
large,  and  thus  SEC-PMC  may  have  been  scored  lower  as  a  consequence. 

It  is  conceivable  that  the  inclusion  of  these  questions  in  the  2007  study  resulted  in  a  differ¬ 
ence  in  the  overall  assessment  of  SEC-PMC  creating  the  observed  relationship  difference. 
Unfortunately,  we  cannot  redo  the  2007  analysis  since  those  data  were  collected  under  a 
promise  of  restricted  use  after  which  they  would  be  discarded. 

4.  SEC-IPT  \n  the  2007  study,  this  relationship  was  evaluated  at  Gamma  =  +0.34.  Now,  it  is 
evaluated  at  Gamma  =  +0.18,  which  is  a  considerable  decrease  from  what  was  a  moderately 
strong  relationship  to  a  weak  one. 

Similar  to  SEC-PP,  the  questions  assessing  SEC-IPT  did  not  change  appreciably  between 
the  two  studies;  however  the  questions  and  process  for  assessing  Perf  did.  To  assess  the  im¬ 
pact  of  this  change,  we  repeated  the  analysis  eliminating  the  additional  questions  and  found 
that  the  analysis  of  the  relationship  between  SEC-PP  and  Perf  did  not  produce  a  result  sub¬ 
stantially  more  consistent  with  the  2007  study.  Hence,  it  is  reasonable  to  claim  that  the 
changes  in  the  assessment  of  Perf  do  not  account  for  the  observed  relationship  difference. 
However  the  explanation  may  lie  in  differences  between  the  two  survey  samples. 

Overall,  there  are  many  more  commonalities  than  differences  in  the  results  of  the  two  surveys.  For 
example,  we  examined  the  impact  of  PC  on  the  relationships  between  various  areas  of  SE  and 
Perf  in  the  2007  study.  The  relationships  were  stronger  in  the  2007  study  for  the  more  challenging 
projects. 

In  the  2012  study,  we  examined  this  impact  to  a  greater  extent  and  found  the  effect  to  hold  tme  in 
all  cases.  Thus,  even  for  the  four  cases  discussed  above,  we  see  that  project  challenge  amplifies 
the  strength  of  the  relationships  between  systems  engineering  and  project  performance  (Figure  25, 
Figure  43,  Figure  49  and  Figure  52). 
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Those  four  relationships  with  process  performance  are  notably  stronger  for  projects  that  are  con¬ 
fronted  by  more  difficult  challenges  (higher  PC).  It  is  the  projects  with  lower  PC  where  the  rela¬ 
tionships  remain  weaker.  That  is  an  encouraging  finding  about  the  validity  of  the  current  results. 

All  statistical  measures  have  noise,  about  which  we  can  only  conjecture  based  on  our  existing  da¬ 
ta.  For  example,  in  our  experience,  the  targeted  organizations  that  were  solicited  for  the  current 
survey  tend  to  have  more  programs  that  regularly  use  advanced  measurement  practices  to  inform 
their  SE  practices.  Other  studies  show  that  such  practices  often  are  accompanied  by  better  project 
performance  and  product  quality  outcomes  [Goldenson  2008,  Stoddard  2010,  McCurley  2010]. 

However,  further  analysis  of  the  existing  data  may  clarify  the  differences  discussed  in  this  section. 
Such  analysis  may  also  augment  the  results  that  are  summarized  in  the  current  report  as  a  whole, 
for  example 

Some  of  the  differences  in  the  strength  of  the  relationships  between  the  SE  variables  and  Perf 
may  be  due  to  the  extent  to  which  the  distributions  of  the  SEC  variables  differ  between  the  two 
samples  with  respect  to  higher  versus  lower  SE  capability. 

If  the  samples  differ,  more  variability  on  the  SEC  variables  may  lead  to  stronger  relationships 
with  Perf. 

Less  variability  on  some  of  the  weighted  summed  indices  also  may  confound  the  results.  Rela¬ 
tionships  may  be  lower  if  there  is  too  little  variability  in  one  or  both  of  the  variables  in  a  statistical 
relationship. 

While  most  of  the  survey  participants  are  working  on  U.S.  DoD  programs,  21%  of  them  do  not. 
That  too  may  affect  the  differences  between  the  two  surveys’  results. 

The  larger  number  of  cases  in  the  current  survey  also  may  account  for  some  of  the  differences  in 
the  strength  of  relationships  between  the  two  studies.  A  few  outliers  can  account  for  more  of  an 
apparent  relationship  in  a  small  sample.  That  is  one  reason  why  statistical  significance  (p-values) 
tends  to  be  lower  with  large  samples. 

The  research  team  will  continue  investigating  these  and  other  possible  reasons  for  any  differences 
in  the  results  of  the  two  surveys.  The  team  will  make  public  the  results  of  those  investigations 
later  rather  than  delay  the  publication  of  the  noteworthy  results  described  in  this  report. 
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7  Summary 


The  impetus  for  this  survey  was  a  desire  to  answer  these  questions: 

What  will  the  application  of  systems  engineering  practices  cost  me? 
What  benefits  will  I  gain  from  the  application  of  these  practices? 


To  address  these  questions,  we  assessed  the  impact  of  the  deployment  of  SE  practices  and  several 
other  factors  on  project  performance.  The  analysis  of  the  collected  data  shows  that  there  are  iden¬ 
tifiable  and  significant  relationships  between  many  of  these  driving  factors  and  project  perfor¬ 
mance.  Figure  68  shows  that  for  projects  deploying  the  least  SE,  only  15%  delivered  higher  pro¬ 
ject  performance,  while  for  those  projects  deploying  the  most  SE,  57%  delivered  higher  project 
performance. 
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Figure  68:  SEC-Total  vs.  Perf 


The  impact  of  SE  is  even  more  apparent  if  we  consider  the  degree  of  challenge  posed  by  the  pro¬ 
ject.  The  chart  on  the  left  side  of  Figure  69  shows  the  relationship  between  SEC-Total  and  Perf 
for  those  projects  presenting  a  lower  degree  of  challenge.  For  these  projects,  the  likelihood  of  de¬ 
livering  higher  project  performance  with  the  least  amount  of  SE  is  a  reasonable  23%.  Flowever, 
the  deployment  of  effective  SE  more  than  doubles  this  likelihood  to  52%. 


Contrast  this  finding  with  the  chart  on  the  right  side  of  Figure  69,  which  shows  the  relationship 
between  SEC-Total  and  Perf  for  those  projects  presenting  a  higher  degree  of  challenge.  Flere,  the 
likelihood  of  delivering  higher  project  performance  without  the  deployment  of  effective  SE  is 
only  8%.  With  the  deployment  of  effective  SE,  this  likelihood  increases  almost  eightfold  to  62%. 
A  Gamma  of  0.62  is  very  strong  for  survey  data  such  as  these.  As  shown  in  Section  5.4  and  later 
in  Section  7,  similarly  strong  relationships  also  exist  for  other  SEC  variables  when  project  chal¬ 
lenge  {PC)  is  high. 
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Figure  69:  SEC-Total  vs.  Perf  Controlled  by  PC 


High  PC 

1  nno/ 

90%  - 

80%  - 

70%  - 

60%  - 

50%  - 

40%  - 

30%  - 

20%  - 

10%  - 

no/T  . 

8% 

26% 

62% 

23% 

35% 

69% 

39% 

12% 

27% 

Lower  SEC  Middle  SEC  Higher  SEC 
(n=26)  (n=23)  (n=26) 

Gamma  =  0.62  p-value  <  0.001 

In  addition  to  looking  at  the  overall  deployment  of  SE  to  projects,  we  examined  the  specific  pro¬ 
cess  groups  of  SE  deployment.  Table  9  shows  the  relationships  between  project  performance  and 
the  deployment  of  SE  in  various  process  groups.  Table  10  shows  the  relationships  between  project 
performance  and  other  factors. 
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Table  9:  Summary  of  Relationships  Between  SE  Deployment  and  Project  Performance 


Driver 

Relationship  to  Performanee  (Gamma) 

All  projeets 

Lower  ehallenge 
projeets 

Higher  challenge 
projects 

SEC-Total  -  total  deployed  SE 

+0.49  =>  Very  strong 
positive 

+0.34^  Strong 
positive 

+0.62  =>  Very  strong 
positive 

SEC-PP  -  project  planning 

+0.46  =>  Very  strong 
positive 

+0.16  ^  Weak 
positive 

+0.65  ^  Very  strong 
positive 

SEC-REQ  -  requirements 
development  and  management 

+0.44  =>  Very  strong 
positive 

+0.36  ^  Strong 
positive 

+0.50  ^  Very  strong 
positive 

SEC-VER  -  verification 

+0.43  =>  Very  strong 
positive 

+0.27  ^  Moderate 
positive 

+0.60  =>  Very  strong 
positive 

SEC-ARCH  -  product  architec¬ 
ture 

+0.41  =>  Very  strong 
positive 

+0.3 1  =>  Moderate 
positive 

+0.49  ^  Very  strong 
positive 

SEC-CM  -  configuration 
management 

+0.38  =>  Strong 
positive 

+0.22  ^  Moderate 
positive 

+0.53  =>  Very  strong 
positive 

SEC-TRD  -  trade  studies 

+0.38  =>  Strong 
positive 

+0.29  ^  Moderate 
positive 

+0.43  ^  Very  strong 
positive 

SEC-PMC  -  project  monitoring 
and  control 

+0.38  =>  Strong 
positive 

+0.27  =>  Moderate 
positive 

+0.53  ^  Very  Strong 
positive 

SEC-VAL  -  validation 

+0.33  =>  Strong 
positive 

+0.23  ^  Moderate 
positive 

+0.48  ^  Very  strong 
positive 

iSE'C-P/-  product  integration 

+0.33  =>  Strong 
positive 

+0.23  =>  Moderate 
positive 

+0.42  ^  Very  strong 
positive 

SEC-RSKM  -  risk  management 

+0.21  =>  Moderate 
positive 

+0.18  ^  Weak 
positive 

+0.24  ^  Moderate 
positive 

SEC-IPT  -  integrated  product 
team  utilization 

+0.18^  Weak 
positive 

-0.12^  Weak 
negative 

+0.40  =>  Very  strong 
positive 

Table  1 0:  Summary  of  Relationships  Between  Other  Factors  and  Project  Performance 


Driver 

Relationship  to  Performance 

All  projects 

Lower  challenge 
projects 

Higher  challenge 
projects 

PC  -  Project  challenge 

-0.26  =>  Moderate 
negative 

-0.26  =>  Moderate 
negative 

-0.23  =>  Moderate 
negative 

EXP  -  Prior  experience 

+0.36  =>  Strong 
positive 

+0.51^  Very  strong 
positive 

+0.19^  Weak 
positive 

The  data  in  Tables  8  and  9  are  illustrated  in  Figures  70  through  72. 
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Figure  70:  Summary  of  Relationships  for  All  Projects 
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Figure  71:  Summary  of  Relationships  for  Less  Challenging  Projects 
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Figure  72:  Summary  of  Relationships  for  More  Challenging  Projects 


The  moderate  and  strong  statistical  relationships  between  systems  engineering  capabilities  and 
project  performance  summarized  in  this  report  are  notable  by  themselves.  Other  things  being 
equal,  projects  deploying  better  systems  engineering  capabilities  tend  to  produce  better  project 
performance.  Of  equal  interest  is  the  impact  of  the  degree  of  challenge  posed  by  the  projects  on 
the  relationships  between  SE  deployment  and  project  performance.  Without  exception,  the  rela¬ 
tionships  between  SE  and  performance  are  stronger  on  more  challenging  projects.  These  results 
can  be  interpreted  as 

For  lower  challenge  projects,  although  SE  improves  the  likelihood  of  project  success,  suc¬ 
cess  remains  somewhat  feasible  without  good  SE  deployment. 

For  higher  challenge  projects,  good  SE  deployment  is  critical  to  project  success. 

The  impact  of  prior  experience  on  project  success  is  also  interesting.  The  success  of  lower  chal¬ 
lenge  projects  is  strongly  related  to  prior  experience.  The  success  of  higher  challenge  projects  is 
less  strongly  related.  One  hypothesis  for  this  difference  is  that  prior  experience  is  helpful  but 
simply  insufficient  to  address  the  difficulties  encountered  in  the  most  challenging  projects.  Deeper 
understanding  of  this  difference  requires  more  information  than  is  available  in  this  study. 

The  information  presented  in  this  report  may  be  used  in  several  ways: 


•  System  developers  can  use  this  knowledge  to  plan  capability  improvement  efforts  for  their  SE 
programs.  By  focusing  improvement  resources  on  those  SE  activities  most  strongly  associat¬ 
ed  with  improved  project  performance,  management  may  optimize  the  efficiency  and  effec¬ 
tiveness  of  those  improvement  efforts. 

•  System  developers  can  use  this  information  as  an  industry  benchmark  against  which  they  can 
compare  their  organization’s  SE  performance.  Projects  within  the  organization  can  be  as¬ 
sessed  in  a  manner  consistent  with  this  study  and  be  compared  with  the  results  of  this  study. 
Weaknesses  can  then  be  improved  or  strengths  applied  with  greater  emphasis  and  leverage. 
The  question-by-question  responses  are  contained  in  the  companion  to  this  report.  As  prom- 
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ised,  survey  participants  have  access  to  the  companion  report  simultaneously  with  the  publi¬ 
cation  of  this  report.  Others  will  not  have  access  to  the  companion  report  until  it  is  publicly 
released  one  year  later. 

•  Systems  engineers  and  SE  managers  at  system  developer  organizations  can  use  this  report  as 
justification  for  and  in  defense  of  their  SE  estimates. 

•  Acquirers  may  use  this  report  to  plan  contractor  evaluations  during  request  for  proposal 
(REP)  development  and  source  selection.  Since  this  survey  shows  clear  statistical  relation¬ 
ships  between  specific  SE  capabilities  and  improved  project  performance,  acquirers  can  struc¬ 
ture  RFPs  and  source  selection  activities  to  include  evaluation  and  consideration  of  these  ca¬ 
pabilities,  which  thereby  may  increase  the  likelihood  of  project  success. 

•  Throughout  the  execution  of  a  project,  acquirers  may  employ  this  survey  or  similar  methods 
to  collect  data  from  suppliers  as  a  means  of  identifying  supplier  deficiencies  contributing  to 
project  risks. 
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8  Next  Steps 


The  BCSE  study  is  a  process  intended  to  produce  a  sustained  and  monitored  improvement  in  the 
acquisition  and  supply  of  systems  through  the  application  of  SE  best  practices.  The  SE  Effective¬ 
ness  Survey  is  the  first  step  in  the  BCSE  Study,  as  noted  in  Figure  73. 
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Jun-2012 
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Phase  III 

ongoing 


Figure  73:  BCSE  Process 


This  report  identifies  a  collection  of  SE  best  practices  that  are  shown  to  have  positive  impact  on 
project  performance.  The  second  phase  of  the  BCSE  study  is  to  promote  action  on  the  findings  of 
this  report. 


Working  through  organizations  such  as  NDIA,  IEEE,  and  INCOSE,  we  can  encourage  the  SE 
community  to  promote  and  build  on  the  findings  of  this  study.  The  SE  community  can  use  the 
findings  to  develop  guidance  for  the  use  of  SE  in  both  acquisition  and  development  programs. 
Researchers  worldwide  can  use  the  findings  to  develop  tools  that  help  systems  engineers  use  these 
best  practices.  Educators  around  the  world  can  develop  training  for  systems  engineers  and  others 
to  promote  the  use  of  these  best  practices.  The  result  of  these  activities  will  be  increased  adoption 
of  SE  best  practices  among  academics,  systems  acquirers,  and  systems  suppliers. 
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The  third  and  final  step  is  to  promote  the  continuous  measurement  of  the  impact  of  SE  best  prac¬ 
tice  adoption.  This  measurement  could  be  done  using  periodic  repeats  of  the  SE  Effectiveness 
Survey,  reverifying  the  effectiveness  of  these  best  practices  and/or  identifying  new  ones. 

The  survey  could  be  administered  easily  within  individual  organizations.  Each  organization  would 
require  that  ongoing  projects  periodically  complete  the  SE  Effectiveness  Survey.  Since  most  or¬ 
ganizations  already  collect  project  performance  data,  the  questionnaire  could  be  abridged  to  just 
the  sections  addressing  SE  work  products.  Completion  of  these  sections  could  be  completed  in  as 
little  as  15  minutes.  The  organizations  could  then  perform  analyses  similar  to  those  defined  in  this 
report  to  monitor  and  track  both  the  deployment  and  effectiveness  of  SE. 

The  ideal  result  would  be  the  establishment  of  a  means  of  continuously  monitoring  both  SE  best 
practice  deployment  and  project  performance  across  many  organizations.  If  the  DoD  chooses  to 
support  this  concept,  it  is  in  a  unique  position  to  support  the  work.  Through  the  Defense  Contract 
Audit  Agency  (DCAA)  and  Defense  Contract  Management  Agency  (DCMA),  the  DoD  already 
collects  a  large  amount  of  data  regarding  project  performance.  The  DoD  could  also  require  sup¬ 
pliers  to  report  on  SE  deployment  by  simply  completing  the  SE  assessment  questions  of  the  SE 
Effectiveness  Survey.  With  the  combination  of  performance  and  SE  deployment  data,  analyses 
similar  to  those  contained  in  this  report  could  be  performed  periodically.  This  process  would  re¬ 
verify  the  effectiveness  of  SE  best  practices  and/or  identify  new  ones,  but  would  also  provide  a 
means  of  tracking  trends  in  SE  deployment  and  project  performance.  All  of  this  information  could 
be  used  to  continually  improve  and  enhance  the  performance  of  development  projects  through  the 
use  of  SE  best  practices. 


CMU/SEI-2012-SR-009  |  81 


CMU/SEI-2012-SR-009  |  82 


Appendix  A  Cross  Reference  to  CMMI 


The  survey  questions  assessing  SE  deployment  were  focused  on  the  production  of  artifacts  from  what  would  be  considered  SE-related  activities.  The  arti¬ 
facts  behind  these  questions  were  based  on  the  CMMI  for  SW/SE/IPPD  vl.l  model,  which  was  current  when  the  earlier  survey  was  developed  [CMMI 
Product  Team  2002,  Elm  2008].  The  specific  artifacts  used  in  the  study  were  chosen  by  a  panel  of  systems  engineers  drawn  from  industry,  government, 
and  academia.  Mapping  of  survey  questions  to  CMMI  is  shown  in  this  appendix.  After  a  wide-ranging  discussion  about  the  nature  of  systems  engineering, 
the  committee  settled  on  CMMI  SW/SE/IPPD  as  the  primary  source  for  the  survey  questions.  Questions  were  derived  from  other  sources  on  occasion  to 
assure  comprehensiveness. 


Table  1 1:  Cross  Reference  to  CMMI 


Process  Area 

Goal 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

Organizational 

Process 

Definition 

SG  1 :  Establish  Organizational 
Process  Assets  -  A  set  of 
organizational  process  assets  is 
established  and  maintained. 

SP  1.1-1:  Establish  Standard  Processes  - 
Establish  and  maintain  the  organization’s 
set  of  standard  processes. 

Organization’s  set  of  standard  processes 

D1 

Project 

Planning 

SG  1 :  Establish  Estimates  - 
Estimates  of  project  planning 
parameters  are  established  and 
maintained. 

SP  1.1-1:  Estimate  the  Scope  of  the  Project 
-  Establish  a  top-level  work  breakdown 
structure  (WBS)  to  estimate  the  scope  of 
the  project. 

Task  descriptions 

D2 

Work  package  descriptions 

D2 

WBS 

D3 

SP  1.2-1:  Establish  Estimates  of  Work 
Product  and  Task  Attributes  -  Establish  and 
maintain  estimates  of  the  attributes  of  the 
work  products  and  tasks. 

Technical  approach 

D6 

SG  2:  Develop  a  Project  Plan  - 
A  project  plan  is  established 
and  maintained  as  the  basis  for 
managing  the  project. 

SP  2.7-1:  Establish  the  Project  Plan  - 
Establish  and  maintain  the  overall  project 
plan  content. 

Integrated  master  plan 

D9 

DIO 

Dll 

Integrated  master  schedule 

D12 

D15 

D16 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

Systems  engineering  management  plan 

D21 

D22 

Systems  engineering  detailed  schedule 

D13 

D14 

GG  2:  Institutionalize  a 

GP  2.7:  Identify  and  Involve  Relevant 

Stakeholders  include  SE  staff 

D4 

Managed  Process 

Stakeholders 

D5 

D7 

D8 

D18 

D19 

SG  1:  Monitor  Project  Against 

SP  1.1-1:  Monitor  Proj  ect  Planning 

Records  of  project  performance 

N1 

Plan  -  Actual  performance  and 
progress  of  the  project  are 
monitored  against  the  project 
plan. 

Parameters  -  Monitor  the  actual  values  of 
the  project  planning  parameters  against  the 
project  plan. 

N2 

N3 

N6 

A14 

Project 

Monitoring  and 
Control 

SP  1.2-1:  Monitor  Commitments  -  Monitor 
commitments  against  those  identified  in  the 
project  plan. 

Records  of  commitment  reviews 

N5 

SP  1.3-1:  Monitor  Project  Risks  -  Monitor 

Records  of  project  risk  monitoring 

F3 

risks  against  those  identified  in  the  project 
plan. 

F4 

F5 

F6 

F7 

SP  1.7-1:  Conduct  Milestone  Reviews  - 
Review  the  accomplishments  and  results  of 
the  project  at  selected  project  milestones. 

Documented  milestone  review  results 

K9 

SG  2:  Manage  Corrective 

SP  2.1-1:  Analyze  Issues  -  Collect  and 

List  of  issues  needing  corrective  actions 

N4 

Action  to  Closure  -  Corrective 

analyze  the  issues  and  determine  the 

06 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

actions  are  managed  to  closure 
when  the  project's  performance 
or  results  deviate  significantly 
from  the  plan. 

corrective  actions  necessary  to  address  the 
issues. 

07 

SP  2.3-1:  Manage  Corrective  Action  - 
Manage  corrective  actions  to  closure. 

Corrective  action  results 

K6 

Integrated 
Project 
Management 
for  IPPD 

SG  4:  Organize  Integrated 

Teams  for  IPPD  -  The 
integrated  teams  needed  to 
execute  the  project  are 
identified,  defined,  structured, 
and  tasked. 

SP  4.1-1:  Determine  Integrated  Team 
Structure  for  the  Project  -  Determine  the 
integrated  team  structure  that  will  best 
meet  the  project  objectives  and  constraints. 

Integrated  team  structures  based  on  the 

WBS  and  adaptations 

El 

E2 

E3 

SP  4.3-1:  Establish  Integrated  Teams  - 
Establish  and  maintain  teams  in  the 
integrated  team  structure. 

Responsibilities  and  authorities  for  each 
integrated  team 

E4 

E5 

Risk 

Management 

SG  2:  Identify  and  Analyze 

Risks  -  Risks  are  identified  and 
analyzed  to  determine  their 
relative  importance. 

SP  2.1-1:  Identify  Risks  -  Identify  and 
document  the  risks. 

List  of  identified  risks,  including  the 
context,  conditions,  and  consequences  of 
risk  occurrence 

FI 

F8 

SG  3 :  Mitigate  Risks  -  Risks 
are  handled  and  mitigated, 
where  appropriate,  to  reduce 
adverse  impacts  on  achieving 
objectives. 

SP  3.1-1:  Develop  Risk  Mitigation  Plans  - 
Develop  a  risk  mitigation  plan  for  the  most 
important  risks  to  the  project,  as  defined  by 
the  risk  management  strategy. 

Risk  mitigation  plans 

F2 

Contingency  plans 

F2 

Requirements 

Management 

SG  1 :  Manage  Requirements  - 
Requirements  are  managed  and 
inconsistencies  with  project 
plans  and  work  products  are 
identified. 

SP  1.1-1:  Obtain  an  Understanding  of 
Requirements  -  Develop  an  understanding 
with  the  requirements  providers  on  the 
meaning  of  the  requirements. 

Lists  of  criteria  for  distinguishing 
appropriate  requirements  providers 

G7 

Criteria  for  evaluation  and  acceptance  of 
requirements 

G8 

An  agreed-to  set  of  requirements 

G9 

SP  1.2-2:  Obtain  Commitment  to 
Requirements  -  Obtain  commitment  to  the 
requirements  from  the  project  participants. 

Requirements  impact  assessments 

GIO 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

SP  1.4-2:  Maintain  Bidirectional 

Traceability  of  Requirements  -  Maintain 
bidirectional  traceability  among  the 
requirements  and  the  project  plans  and 
work  products. 

Requirements  tracking  system 

G12 

GG  2:  Institutionalize  a 

GP  2.6:  Manage  Configurations 

Configuration  management  records 

G13 

Managed  Process 

GP  2.7:  Identify  and  Involve  Relevant 
Stakeholders 

<Stakeholders  include  SE  staff> 

G14 

SG  1 :  Develop  Customer 
Requirements  -  Stakeholder 
needs,  expectations, 
constraints,  and  interfaces  are 
collected  and  translated  into 
customer  requirements. 

SP  1.1-1:  Collect  Stakeholder  Needs  - 
Identify  and  collect  stakeholder  needs, 
expectations,  constraints,  and  interfaces  for 
all  phases  of  the  product  lifecycle. 

Customer  requirements 

Gil 

SP  1.2-1:  Develop  the  Customer 
Requirements  -  Transform  stakeholder 
needs,  expectations,  constraints,  and 
interfaces  into  customer  requirements. 

Customer  requirements 

G1 

Requirements 

Development 

SG  2:  Develop  Product 
Requirements  -  Customer 
requirements  are  refined  and 
elaborated  to  develop  product 
and  product-component 

SP  2.1-1:  Establish  Product  and  Product- 
Component  Requirements  -  Establish  and 
maintain  product  and  product-component 
requirements,  which  are  based  on  the 
customer  requirements. 

Derived  requirements 

G2 

requirements. 

SP  2.2-1:  Allocate  Product-Component 
Requirements  -  Allocate  the  requirements 
for  each  product  component. 

Requirement  allocation  sheets 

G3 

SG  3:  Analyze  and  Validate 

SP  3.1-1:  Establish  Operational  Concepts 

Operational  concept 

G4 

Requirements  -  The 
requirements  are  analyzed  and 
validated,  and  a  definition  of 
required  functionality  is 
developed. 

and  Scenarios  -  Establish  and  maintain 
operational  concepts  and  associated 

Product  installation,  operational, 
maintenance,  and  support  concepts 

G6 

scenarios. 

Use  cases 

G5 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

Technical 

Solution 

SG  1 :  Select  Product- 
Component  Solutions  -  Product 
or  product-component 
solutions  are  selected  from 
alternative  solutions. 

SP  1.1-1:  Develop  Alternative  Solutions 
and  Selection  Criteria  -  Develop  alternative 
solutions  and  selection  criteria. 

Alternative  solutions 

H2 

Selection  criteria 

H2 

SP  1.3-1:  Select  Product-Component 
Solutions  -  Select  the  product-component 
solutions  that  best  satisfy  the  criteria 
established. 

Product-component  selection  decisions  and 
rationale 

H3 

Documented  solutions,  evaluations,  and 
rationale 

H3 

SG  2:  Develop  the  Design  - 
Product  or  product-component 
designs  are  developed. 

SP  2.1-1:  Design  the  Product  or  Product 
Component  -  Develop  a  design  for  the 
product  or  product  component. 

Product  architecture 

13 

14 

SP  2.3-1:  Establish  Interface  Descriptions  - 
Establish  and  maintain  the  solution  for 
product-component  interfaces. 

Interface  design 

11 

Interface  design  documents 

11 

SP  2.3-3:  Design  Interfaces  Using  Criteria  - 
Design  comprehensive  product-component 
interfaces  in  terms  of  established  and 
maintained  criteria. 

Interface  control  documents 

12 

GG  2:  Institutionalize  a 

Managed  Process 

GP  2.7:  Identify  and  Involve  Relevant 
Stakeholders 

<Stakeholders  include  SE  staff> 

HI 

15 

Product 

Integration 

SG  1 :  Prepare  for  Product 
Integration  -  Preparation  for 
product  integration  is 
conducted. 

SP  1.3-3:  Establish  Product  Integration 
Procedures  and  Criteria  -  Establish  and 
maintain  procedures  and  criteria  for 
integration  of  the  product  components. 

Product  integration  procedures 

J1 

Verification 

SG  I:  Prepare  for  Verification  - 
Preparation  for  verification  is 
conducted. 

SP  1.3-3:  Establish  Verification  Procedures 
and  Criteria  -  Establish  and  maintain 
verification  procedures  and  criteria  for  the 
selected  work  products. 

Verification  procedures 

K1 

V erification  criteria 

K2 

SG  2:  Perform  Peer  Reviews  - 
Peer  reviews  are  performed  on 
selected  work  products. 

SP  2.1-1:  Prepare  for  Peer  Reviews  - 
Prepare  for  peer  reviews  of  selected  work 
products. 

Peer  review  schedule 

D17 

Entry  and  exit  criteria  for  work  products 

K3 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

Peer  review  training  material 

K4 

Selected  work  products  to  be  reviewed 

K5 

SP  2.2-1:  Conduct  Peer  Reviews  -  Conduct 
peer  reviews  on  selected  work  products  and 
identify  issues  resulting  from  the  peer 

Peer  review  issues 

K7 

review. 

K9 

SP  2.3-2:  Analyze  Peer  Review  Data  - 
Analyze  data  about  preparation,  conduct, 
and  results  of  the  peer  reviews. 

Peer  review  action  items 

K6 

Validation 

SG  1:  Prepare  for  Validation  - 

SP  1.3-3:  Establish  Validation  Procedures 

Validation  procedures 

LI 

Preparation  for  validation  is 
conducted. 

and  Criteria  -  Establish  and  maintain 
procedures  and  criteria  for  validation. 

Validation  criteria 

L2 

SG  1 :  Establish  Baselines  - 
Baselines  of  identified  work 
products  are  established. 

SP  1.1-1:  Identify  Configuration  Items  - 
Identify  the  configuration  items, 
components,  and  related  work  products  that 
will  be  placed  under  configuration 
management. 

Identified  configuration  items 

Ml 

Configuration 

Management 

SP  1.2-1:  Establish  a  Configuration 
Management  System  -  Establish  and 
maintain  a  configuration  management  and 
change  management  system  for  controlling 
work  products. 

Configuration  management  system  access 
control  procedures 

M2 

SP  1.3-1:  Create  or  Release  Baselines  - 

Baselines 

M4 

Create  or  release  baselines  for  internal  use 
and  for  delivery  to  the  customer. 

N1 

SG  2:  Track  and  Control 

SP  2.2-1:  Control  Configuration  Items  - 

Revision  history  of  configuration  items 

M3 

Changes  -  Changes  to  the  work 
products  under  configuration 
management  are  tracked  and 
controlled. 

Control  changes  to  the  configuration  items. 

Archives  of  the  baselines 

M3 
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Process  Area 

Goal  I 

PRACTICE 

WORK  PRODUCT 

SURVEY 

QUESTION 

SG  3 :  Establish  Integrity  - 
Integrity  of  baselines  is 
established  and  maintained. 

SP  3.2-1 :  Perform  Configuration  Audits  - 
Perforiu  configuration  audits  to  maintain 
integrity  of  the  configuration  baselines. 

Configuration  audit  results 

K8 
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Appendix  B  Questionnaire 


Software 

The  Effectiveness 
A  Survey 

The  National  Defense  Industrial  Association  (NDIA),  the  IEEE  Aerospace  and  Electronic 
Systems  Society  (lEEE-AESS)  and  the  Software  Engineering  Institute  (SEI)  welcome  you 
to  your  personalized  questionnaire  for  our  survey  on  “The  Effectiveness  of  Systems  Engi¬ 
neering.”  Our  hope  is  that  your  participation  will  help  your  project  and  organization  evaluate 
the  effectiveness  of  their  Systems  Engineering  practices  relative  to  the  successes  and  chal¬ 
lenges  reported  by  others  throughout  the  industry. 

Most  of  the  information  necessary  to  complete  the  questionnaire  should  be  easily  accessible 
or  familiar  to  you  or  perhaps  an  informed  designee.  It  should  take  about  30  to  45  minutes  to 
complete  the  questionnaire.  Please  provide  your  best  estimates  if  quantitative  measurements 
are  unavailable. 

Please  complete  the  questionnaire  as  candidly  and  completely  as  you  possibly  can.  The  re¬ 
sults  will  be  useful  to  you,  us  and  others  only  to  the  extent  that  all  survey  participants  do  so. 
There  is  no  need  to  hide  weaknesses  or  embellish  strengths.  Remember  that  your  response 
will  be  anonymous.  Neither  the  SEI  nor  anyone  else  will  know  the  person,  project,  or  organ¬ 
ization  reflected  in  your  response.  The  information,  collected  under  promise  of  non  disclosure 
by  the  SEI,  will  be  held  in  strict  confidence  and  will  not  be  released  in  any  manner.  Survey 
results  will  be  reported  only  in  summary  aggregate  form.  Individual  responses  will  NOT  be 
exposed.  No  attribution  to  people,  projects,  or  organizations  will  be  made. 

A  detailed  summary  report  of  the  survey  results  will  be  prepared  by  the  SEI.  The  report  will 
provide  a  baseline  against  which  you  can  compare  the  performance  of  your  project  and  or¬ 
ganization.  As  a  reward  for  participating  in  this  survey,  the  report  will  be  initially  released  only 
to  those  who  fully  complete  a  survey  questionnaire.  The  report  will  be  not  be  publicly  re¬ 
leased  until  one  year  later. 

Thank  you  once  again  for  your  help  with  this  important  activity.  Please  feel  free  to  contact  us 
at  sei-analvsis@sei.cmu.edu  if  you  have  any  difficulty  with  the  questionnaire. 


Engineering  Institute  |  ( jarno^McIMel Ion  ^ IEEE ivf 

of  Systems  Engineering: 
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CHARACTERIZATION  OF  THE  PROJECT,  PRODUCT,  CONTRACT,  AND 

ORGANIZATION 


A.  ABOUT  THIS  PROJECT 


The  information  gathered  here  and  in  the  next  few  sections  wiii  be  used  by  the  survey  ana- 
iysts  to  categorize  the  participating  projects  and  organizations  in  order  to  better  understand 
the  responses  to  subsequent  questions  about  systems,  Systems  Enqineerinq  practices,  and 
project  performance. 

The  terms  "Project",  and  "Proqram",  are  used  interchangeabiy  throughout  this  survey.  Both 
refer  to  any  temporary  endeavor,  having  a  defined  beginning  and  end,  undertaken  to  meet 
unique  goais  and  objectives.  Such  endeavors  are  characterized  by  a  defined  set  of  objec¬ 
tives,  a  defined  budget  or  cost  estimate,  and  a  defined  scheduie  or  period  of  performance. 

In  crafting  your  response  to  this  survey,  it  is  important  that  you  keep  in  mind  a  ciear  idea  of 
the  scope  of  the  project  for  which  you  are  responding.  This  wiii  heip  to  ensure  that  your  re¬ 
sponses  regarding  appiied  Systems  Engineering  activities  and  your  responses  regarding  pro¬ 
ject  performance  reiate  to  the  same  body  of  work. 

Foiiowing  are  severai  statements  that  have  been  used  to  characterize  various  deveiopment 
projects.  How  weii  do  the  statements  describe  this  project? 

1 .  The  project  is  chaiienging  because  there  is  no  precedent  for  what  is  being  done.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  is  chaiienging  because  significant  constraints  are  piaced  on  the  quaiity  attrib¬ 
utes  (e.g.  reiiabiiity,  scaiabiiity,  security,  supportabiiity,  etc.)  of  the  product.  (Please  select 
one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  The  project  is  chaiienging  because  the  size  of  the  deveiopment  effort  is  iarge.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  The  project  is  chaiienging  because  the  technoiogy  needed  for  this  project  is  not  mature  or 
otherwise  poses  a  high  risk.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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5.  The  project  is  challenging  because  there  are  extensive  needs  for  interoperability  with  oth¬ 
er  systems  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

6.  The  project  is  challenging  because  there  are  insufficient  resources  (e.g.  people,  funding) 
available  to  support  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

7.  The  project  is  challenging  because  there  are  insufficient  skills  and  subject  matter  exper¬ 
tise  available  to  support  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  The  project  is  challenging  for  other  reasons  (Please  describe  briefly) 


9.  In  the  past,  this  project  team  has  successfully  completed  projects  of  similar  scope. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

10.  The  requirements  supplied  by  the  customer  for  this  project  are  well-defined  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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1 1 .  The  requirements  supplied  by  the  customer  for  this  project  have  not  changed  sufficiently 
to  generate  a  significant  impact  on  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  What  percentage  of  the  customer  technical  requirements  were  marked  “To  Be  Deter¬ 
mined”  or  equivalent  at  time  of  contract  award?  (Please  specify  -  numbers  only,  without 
the  percentage  sign) 


13.  What  percentage  of  the  customer’s  technical  requirements  are  currently  marked  “To  Be 
Determined”  or  equivalent?  (Please  specify  an  approximate  percentage  -  without  the 
percentage  sign) 


14.  Do  you  separately  budget  and  track  Systems  Enqineerinq  activities?  (Please  select  one) 
O  Yes 

O  No 

O  Don't  Know 

15.  Approximately  what  percentage  of  non-recurrinq  enqineerinq  (NRE)  does  Systems  Enqi¬ 
neerinq  represent?  (Please  specify  an  approximate  percentage  -  without  the  percentage 
sign) 


16.  How  are  Systems  Enqineerinq  activities  estimated  and  budgeted?  (Please  check  all  that 
apply) 

□  They  are  budgeted  as  a  percentage  of  the  total  development  cost 

□  They  are  estimated  using  a  parametric  cost  model  (e.g.,  COSYSMO,  SEER,  SLIM, 
TruePlanning) 

□  They  are  estimated  on  a  task-by-task  basis 

□  Other  (please  describe) 

I - 3 
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17.  Which  of  the  following  best  describes  the  ultimate  end-user  of  this  product?  (Please  se¬ 
lect  one) 

O  Government  (USA)  -  defense  related 
O  Government  (USA)  -  not  defense  related 
O  Government  (non-USA)  -  defense  related 
O  Government  (non-USA)  -  not  defense  related 
O  Industrial  /  Commercial 
O  Private  Consumer 
O  Other  (Please  describe) 
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B.  ABOUT  THE  CONTRACT 

1 .  What  is  the  current  total  contract  value  of  this  project?  (Please  specify  in  US  dollars  - 
numbers  only,  without  a  dollar  sign  or  commas) 

I  US  dollars  ($) 

2.  What  was  the  initial  contract  value  of  this  project?  (Please  specify  in  US  dollars  —  num¬ 
bers  only,  without  a  dollar  sign  or  commas) 

I  US  Dollars  ($) 

3.  The  change  in  contract  value  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  contract  value  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  cost  of  the  project 
O  Other  (please  explain) 

- 3 

4.  What  is  the  current  total  planned  duration  of  this  project  or  contract?  (Please  specify  in 
months  -  numbers  only) 

I  Calendar  months 

5.  What  was  the  initial  total  planned  duration  of  this  project  or  contract?  (Please  specify  in 
months  -  numbers  only) 

I  Calendar  months 

6.  The  change  in  schedule  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  schedule  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  schedule  for  executing  the  project 
O  Customer  driven  increases  in  the  schedule  for  executing  the  project 
O  Other  (please  explain) 

- 3 

3J  3^ 

7.  What  was  the  initial  total  budget  for  this  project? 

I  US  dollars  ($) 


CMU/SEI-2012-SR-009  |  96 


8.  What  is  the  current  total  budget  for  this  project? 


I  US  dollars  ($) 

9.  The  change  in  budget  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  budget  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  cost  of  executing  the  project 
O  Customer  driven  increases  in  the  cost  of  executing  the  project 
O  Other  (please  explain) 

- 3 

3J  3^ 

10.  How  many  contract  change  orders  have  been  received?  (Please  specify  a  number,  ap¬ 
proximate  If  necessary) 

I  Change  orders 

11.  This  contract,  includes  provisions  for  additional  payments  based  on  meeting  or  exceeding 
cost,  schedule,  and/or  performance  targets  (e.g.,  incentive  fees,  award  fees).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  What  is  the  current  completion  status  of  this  project?  (Please  specify  an  approximate 
percentage  -  without  the  percentage  sign  -  e.g.,  60  for  a  project  that  Is  60%  complete) 

I  %  Complete 

13.  What  type  of  contract(s)  was  awarded  for  this  project?  (Please  select  one) 

O  This  is  a  fixed-price  contract  -  the  total  contract  value  is  primarily  determined  by  the  ini¬ 
tial  contract,  (e.g.,  FFP,  FPIF,  FFP-LOE). 

O  This  is  a  cost-reimbursable  contract  -  the  total  contract  value  is  primarily  determined  by 
my  cost  of  executing  the  contract  (e.g.,  CPFF,  CPAF,  CPIF). 

O  This  contract  does  not  fit  the  categories  listed  above.  (Please  describe) 

- 3 

3J  3^ 
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C.  ABOUT  THE  ORGANIZATION 

By  "organization"  we  mean  an  administrative  structure  within  which  (possibiy  many)  projects 
or  simiiar  work  efforts  are  organized  under  common  management  and  poiicies. 

When  thinking  about  your  organization,  piease  answer  for  the  unit  to  which  this  project  re¬ 
ports  administrativeiy,  e.g.,  a  site,  division  or  department,  not  for  a  iarger  enterprise  of  which 
the  organization  to  which  you  report  may  be  a  part. 

Foiiowing  are  severai  statements  that  have  been  used  to  characterize  various  deveiopment 
organizations.  How  weii  do  the  statements  describe  this  project's  parent  organization? 

1.  This  organization  has  successfuiiy  compieted  projects  simiiar  in  scope  to  this  one  in  the 
past.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  Within  this  organization  ...  (Please  select  one) 

O  Systems  Engineering  skiiis  and  responsibiiities  are  contained  in  a  separate  department. 

O  Systems  Engineering  skiiis  and  responsibiiities  are  distributed  throughout  other  de¬ 

partments. 


3.  Which  of  these  best  describes  your  industry  or  service?  (Please  select  one) 

O  Industriai  Manufacturing  and  Services  -  Aerospace  and  Defense 

O  Industriai  Manufacturing  and  Services  -  Eiectronic  and  Eiectricai  Equipment 

O  Industriai  Manufacturing  and  Services  -  Other  (please  specify) 


3 

o 

Transportation 

o 

Energy 

o 

Communications 

o 

Consumer  Goods  and  Services 

o 

Heaith  Care 

o 

Other  (please  specify) 

jU 

4.  Piease  enter  the  country  in  which  most  of  the  design  and  deveiopment  engineering  wiii 
be/was  performed. 
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5.  Is  anything  else  particularly  important  in  characterizing  your  project,  product,  contract,  or 
organization  within  which  it  resides.  (Please  describe  here) 
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SYSTEMS  ENGINEERING  CAPABILITIES  ASSESSMENT 

This  and  the  next  few  sections  ask  you  about  the  Systems  Engineering  activities  performed 
on  this  project.  Most  of  the  questions  ask  about  the  existence  and  quaiity  of  tangibie  work 
products.  Note  that  the  pertinent  information  often  may  be  distributed  throughout  muitipie 
documents  or  other  work  products;  it  need  not  necessariiy  be  iocated  in  one  particuiar  piace. 

Foiiowing  are  severai  statements  about  work  products  and  activities  that  are  sometimes  used 
for  systems  deveiopment.  Piease  use  the  foiiowing  definitions  to  describe  their  use  on  this 
project: 

Strongly  Disagree:  The  work  product  does  not  exist  or  is  never  used  on  this  project. 

Disagree:  The  work  product  is  of  insufficient  quaiity  or  is  not  used  reguiariy  at  appropriate 
occasions  on  this  project. 

Agree:  The  work  product  or  practice  is  of  good  quaiity  and  it  is  used  reguiariy  on  this  project, 
aithough  not  necessariiy  as  often  as  it  couid  be. 

Strongly  Agree:  The  work  product  or  practice  is  of  exceptionai  quaiity  and  it  is  used  at  neariy 
aii  appropriate  occasions  on  this  project. 


D.  PROJECT  PLANNING 

1 .  This  project  utiiizes/utiiized  a  documented  set  of  Systems  Enqineerinq  processes  for  the 
pianning  and  execution  of  the  project.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
inciuded  task  descriptions  and  work  package  descriptions.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
was  based  upon  the  product  structure.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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4.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
was  developed  with  the  active  participation  of  those  who  perform  the  systems  engineer¬ 
ing  activities.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
was  developed  and  maintained  with  the  active  participation  of  all  relevant  stakeholders 
(e.g.,  developers,  maintainers,  testers,  inspectors,  etc.).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

6.  This  project’s  Technical  Approach  is  complete,  accurate  and  up-to-date.  (Please  select 
one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

7.  This  project’s  Technical  Approach  is  developed  and  maintained  with  the  active  participa¬ 
tion  of  those  who  perform  the  Systems  Engineering  activities.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  This  project’s  Technical  Approach  is  developed  and  maintained  with  the  active  participa¬ 
tion  of  all  appropriate  functional  stakeholders.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

9.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  an 
event-driven  plan  (i.e.,  each  accomplishment  is  tied  to  a  key  project  event).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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10.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  docu¬ 
ments  significant  accomplishments  with  pass/fail  accomplishment  criteria  for  both  busi¬ 
ness  and  technical  elements  of  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

11.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  con¬ 
sistent  with  the  Work  Breakdown  Structure  (WBS).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  This  project  has  an  integrated  event-based  schedule  that  is  structured  as  a  networked, 
multi-layered  schedule  of  project  tasks  reguired  to  complete  the  work  effort.  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

13.  This  project  has  an  integrated  event-based  schedule  that  contains  a  compilation  of  key 
technical  accomplishments  (e.g.,  a  Systems  Engineering  Master  Schedule).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

14.  This  project  has  an  integrated  event-based  schedule  that  references  measurable  criteria 
(usually  contained  in  the  Integrated  Master  Plan)  reguired  for  successful  completion  of 
key  technical  accomplishments.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

15.  This  project  has  an  integrated  event-based  schedule  that  is  consistent  with  the  Work 
Breakdown  Structure  (WBS).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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16.  This  project  has  an  integrated  event-based  schedule  that  identifies  the  critical  path  of  the 
program  schedule.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

17.  This  project  has  a  plan  or  plans  for  the  performance  of  technical  reviews  with  defined  en¬ 
try  and  exit  criteria  throughout  the  life  cycle  of  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

1 8.  The  Systems  Engineering  function  actively  participates  in  the  development  and  updates 
of  the  project  planning.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

19.  Those  who  perform  Systems  Engineering  activities  actively  participate  in  track¬ 
ing/reporting  of  task  progress.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

20.  The  acguirer  provided  this  project  with  a  Systems  Engineering  Plan  in  a  timely  manner. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

21 .  This  project  has  a  plan  or  plans  that  include  details  of  the  management  of  the  integrated 
technical  effort  across  the  project  (e.g.,  a  Systems  Engineering  Management  Plan  or  a 
Systems  Engineering  Plan).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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22.  The  Systems  Engineering  Management  Plan  (or  equivalent)  developed  by  the  project 
team  is  aligned  and  consistent  with  the  Systems  Engineering  Plan),  (or  equivalent)  pro¬ 
vided  by  the  acquirer.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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E.  INTEGRATED  PRODUCT  TEAMS 

1 .  This  project  makes  effective  use  of  integrated  product  teams  (IPTs).  (Please  select  one) 
O  Strongiy  disagree 

O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  My  acquirer  participates  in  my  integrated  product  teams  (IPTs)  for  this  project.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  My  suppiiers  activeiy  participate  in  my  integrated  product  teams  (IPTsj.  (Please  select 
one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  an  integrated  product  team  (IPTsj  with  assigned  responsibiiity  for  Sys¬ 
tems  Engineering.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  has  Systems  Engineering  representation  on  each  integrated  product  teams 
(IPTs).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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F.  RISK  MANAGEMENT 

1 .  This  project  has  a  Risk  Management  process  that  creates  and  maintains  an  accurate  and 
up-to-date  iist  of  risks  affecting  the  project  (e.g.,  risks  to  cost,  risks  to  scheduie,  risks  to 
performance)  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  a  Risk  Management  process  that  creates  and  maintains  up-to-date  doc¬ 
umentation  of  risk  mitigation  pians  and  contingency  pians  for  seiected  risks  (Please  se¬ 
lect  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has  a  Risk  Management  process  that  monitors  and  reports  the  status  of  risk 
mitigation  activities  and  resources.  ((Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  a  Risk  Management  process  that  assesses  risk  against  achievement  of 
an  event-based  scheduie  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project's  Risk  Management  process  is  integrated  with  project  decision-making. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project's  Risk  Management  process  is  integrated  with  program  cost  and/or  earned 
vaiue  management.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project's  Risk  Management  process  is  integrated  with  program  scheduiing  (e.g., 
risks  are  incorporated  in  the  program  master  scheduies).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project's  Risk  Management  process  integrates  subcontract  or  suppiier  risk  manage¬ 
ment  processes.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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G.  REQUIREMENTS  DEVELOPMENT  AND  MANAGEMENT 

1.  This  project  maintains  an  up-to-date  and  accurate  iisting  of  aii  requirements  specified  by 
the  customer,  to  inciude  reguiatory,  statutory,  and  certification  requirements.  (Please  se¬ 
lect  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  maintains  an  up-to-date  and  accurate  iisting  of  aii  requirements  derived  from 
those  specified  by  the  customer.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  maintains  up-to-date  and  accurate  documentation  cieariy  reflecting  the  hier- 
archicai  aiiocation  of  both  customer  and  derived  requirements  to  each  eiement  (subsys¬ 
tem,  component,  etc.)  of  the  system  in  the  configuration  baseiines.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  operation- 
ai  concepts  and  their  associated  scenarios.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  use  cases 
(or  their  equivaient).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  product 
instaiiation,  maintenance  and  support  concepts.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project  has  documented  criteria  for  identifying  authorized  requirements  providers  to 
avoid  requirements  creep  and  voiatiiity.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project  has  documented  criteria  (e.g.,  cost  impact,  scheduie  impact,  authorization  of 
source,  contract  scope,  requirement  quaiity)  for  evaiuation  and  acceptance  of  require¬ 
ments.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

9.  The  requirements  for  this  project  are  approved  in  a  formai  and  documented  manner  by 
reievant  stakehoiders.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

10.  This  project  performs  and  documents  requirements  impact  assessments  for  proposed 
requirements  changes  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

11.  This  project  deveiops  and  documents  project  requirements  based  upon  stakehoider 
needs,  expectations,  and  constraints.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

12.  This  project  has  an  accurate  and  up-to-date  requirements  manaqement  system.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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13.  For  this  project,  the  requirements  documents  are  managed  under  a  configuration  controi 
process.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

14.  For  this  project,  the  requirements  documents  are  accessibie  to  aii  reievant  project  staff. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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H.  TRADE  STUDIES 

I .  Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and  perfor¬ 
mance  of  those  trade  studies.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

2.  This  project  performs  and  documents  trade  studies  between  alternate  solutions  in  a  time¬ 
ly  manner,  and  based  upon  definitive  and  documented  selection  criteria.  (Please  select 
one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

3.  Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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I.  PRODUCT  ARCHITECTURE 

1.  This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  controi  docu¬ 
ments,  modeis,  etc.)  defining  interfaces  in  detaii.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  Interface  definition  descriptions  are  maintained  in  a  designated  iocation,  under  configura¬ 
tion  management,  and  accessibie  to  aii  who  need  them.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  For  this  project,  the  product  high-ievei  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  controi.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  For  this  project,  the  product  high-ievei  structure  is  documented  using  muitipie  views  (e.g. 
functionai  views,  moduie  views,  etc.).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  For  this  project,  the  product  high-ievei  structure  is  accessibie  to  aii  reievant  project  per- 
sonnei.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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J.  PRODUCT  INTEGRATION 

1.  This  project  has  accurate  and  up-to-date  documents  defining  its  product  integration  pro¬ 
cess,  pians,  criteria,  etc.  throughout  the  iife  cycie.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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K.  VERIFICATION 

1 .  This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
test  and  verification  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  verification  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  for  work  products  that  defines  entry  and  exit  criteria.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  inciudes  training  the  reviewers  to  conduct  reviews.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  criteria  for  the  seiection  of  work  products  (e.g.,  requirements 
documents,  test  pians,  system  design  documents,  etc.)  for  review.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  tracks  action  items  to  ciosure.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  addresses  identified  risks  and  risk  mitigation  activities  during  reviews. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  examines  compieteness  of  configuration  baseiines.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

9.  This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  quaiified  personnei  with  no 
connection  to  or  stake  in  the  project)  and  documents  resuits,  issues,  action  items,  risks, 
and  risk  mitigations  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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L.  VALIDATION 

1 .  This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
vaiidation  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  vaiidation  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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M.  CONFIGURATION  MANAGEMENT 

1.  This  project  maintains  a  iisting  of  items  managed  under  configuration  controi.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  a  configuration  management  system  that  charters  a  Change  Controi 
Board  to  disposition  change  requests.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  maintains  records  of  requested  and  impiemented  changes  to  configuration- 
managed  items.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  creates  and  manages  configuration  baseiines  (e.g.,  functionai,  aiiocated, 
product).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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N.  PROJECT  PERFORMANCE:  EARNED  VALUE  MANAGEMENT 

1.  This  project  creates  and  manages  cost  and  schedule  baselines.  (Please  select  one) 

O  Strongly  disagree 

O  Disagree 
O  Agree 
O  Strongly  Agree 

2.  Earned  Value  Management  System  (EVMSj  data  are  available  to  decision  makers  in  a 
timely  manner  (i.e.  current  within  2  weeks).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

3.  The  requirement  to  track  and  report  Earned  Value  Management  System  (EVMS)  data  is 
levied  upon  the  project’s  suppliers.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

4.  Variance  thresholds  for  the  Cost  Performance  Index  (CPI)  and  Schedule  Performance 
Index  (SPI)  are  defined,  documented,  and  used  to  determine  when  corrective  action  is 
needed.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  The  Earned  Value  Management  System  (EVMS)  is  linked  to  the  technical  effort  through 
the  Work  Breakdown  Structure  (WBS),  the  Integrated  Master  Plan  (IMP),  (or  equivalent), 
and  the  Integrated  Master  Schedule  (IMS)  (or  equivalent).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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6.  When  is  the  Earned  Value  Management  System  (EVMS)  baseline  updated?  (Please  se¬ 
lect  as  many  as  apply) 

□  Only  at  contract  initiation 

□  Whenever  a  contract  change  order  or  renewal  is  received 

□  Incrementally  in  rolling  wave  planning 

□  Whenever  the  project  is  reprogrammed  due  to  a  pre-determined  cost  or  schedule  vari¬ 
ance 

□  At  periodic  intervals 

□  Other  (Please  describe  briefly) 


:3 


7.  What  is  the  projected  cost  variance  at  completion  for  the  current  contract  baseline? 
(Please  specify  an  amount  In  US  Dollars  ($),  using  +  signs  for  any  overruns  and  -  signs 
for  any  underruns) 


US  Dollars  ($) 


8.  What  is  the  projected  schedule  variance  at  completion  for  the  current  contract  baseline? 
(Please  specify  In  months,  using  +  signs  for  any  late  delivery  and  -  signs  for  early  deliv¬ 
ery) 


Duration  in  months 


9.  What  is  the  current  cumulative  (or  final)  EVMS  Cost  Performance  Index  (CPI)  for  this  pro¬ 
ject?  (Please  specify  a  number) 


10.  What  is  the  current  cumulative  (or  final)  EVMS  Schedule  Performance  Index  (SPI)  for  this 
project?  (Please  specify  a  number) 
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O.  OTHER  PERFORMANCE  INDICATORS 

1 .  What  percentage  of  available  Award  Fees  have  been  received  by  this  project  in  the  cur¬ 
rent  period  of  performance?  (Please  specify  an  approximate  percentage  -  without  the 
percentage  sign.  Enter  "n/a"  if  this  contract  does  not  include  Award  Fees.) 


2.  What  percentage  of  available  Award  Fees  have  been  received  by  this  project  to  date  (i.e., 
in  all  periods)?  (Please  specify  an  approximate  percentage  -  without  the  percentage 
sign.  Enter  "n/a"  if  this  contract  does  not  include  Award  Fees.) 


3.  Requirements  are  being  satisfied  and  remain  on  track  to  be  satisfied  in  the  product  re¬ 
leases  as  originally  planned;  they  are  not  being  deleted  or  deferred  to  later  releases. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

4.  Overall,  this  project  is  performing  per  the  schedule  established  in  the  current  Integrated 
Master  Schedule  (IMS)  approved  by  the  acquirer.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  The  schedule  of  this  project’s  critical  path,  when  compared  to  the  current  Integrated  Mas¬ 
ter  Schedule  (IMS)  approved  by  the  acquirer  is  ...  (Please  select  one) 

O  Greater  than  6  months  late 
O  3  to  6  months  late 
O  1  to  3  months  late 
O  Within  plus  or  minus  1  month 
O  1  to  3  months  early 
O  3  to  6  months  early 

6.  This  project  collects  and  tracks  (or  will  collect  and  track)  reports  of  problems  from  fielded 
items.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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7.  This  project  conducts  (or  will  conduct)  engineering  assessments  of  all  field  trouble  re¬ 
ports.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  the 
schedule.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

9.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  cost. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

10.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  sat¬ 
isfaction  of  requirements.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

11.  What  performance  indicators  (beyond  cost  and  schedule)  have  been  particularly  useful 
for  managing  your  project?  (Please  describe  here) 

- 3 

3J  3^ 

12.  What  other  kinds  of  performance  related  information  would  have  been  helpful  for  your 
project  or  program,  but  was  unavailable?  (Please  describe  here) 


“31 
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13.  What  indicators  do  you  use  in  your  project  or  organization  to  determine  Systems  Engi¬ 
neering  effectiveness?  (Please  describe  here) 


14.  What  indicators  of  Systems  Engineering  effectiveness  are  reguiariy  reviewed  across  pro¬ 
jects  by  higher  ievei  management?  (Please  describe  here) 
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P.  IN  CONCLUSION 

1 .  Is  there  anything  else  that  you  would  like  to  tell  us  about  your  project  or  this  survey? 
(Please  describe  here) 


Thank  you  very  much  for  your  time  and  effort! 


Please  be  sure  to  use  the  Save  button.  That  will  take  you  to  the  final  page  where  you  may 
SUBMIT  your  response. 


Copyright  2012,  Carnegie  Mellon  University. 
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Appendix  C  Solicitation  of  an  Invitation  to  Respondents 


Sample  solicitation  email 


Business  Case  for  Systems  Engineering 

From:  Business  Cdse  for  Systems  Engineering 
Senb  WednescUy,  October  26,  2011  11:17  AM 
To:  cnemfaer's  etrual  dddress> 

Subject:  NDIA  /  IEEE  Systems  Engineering  Sfecbveness  Sixty  -  Pdrbdpdbon  inquiry  Acces50dtdColecticn:{< Access  ID 
Code>} 

NOTE;  This  study  is  being  performed  by  the  Nabonal  Defense  Industrial  Association  (NDIA)  and  the  IEEE  Aerospace  and 
Electronic  Systems  Society  (lEEE-AESS).  If  you  are  a  member  of  more  than  one  of  these  organizations,  you  may  receive 
multiple  copies  of  this  email.  We  apologize  for  any  irtconvenienoe. 

Dear  Coleague 

The  Systems  Engineering  Efiectiveness  Committee  —  a  joint  committee  of  the  National  Defense  Industrial  Association  (NDIA). 
the  IEEE  Aerospace  and  Electronic  Systems  Society  (lEEE-AESS).  arxl  the  Software  Engirieering  Institute  (SEI)  is  conducting  a 
study  of  Systems  Engineering  Effectiveness  to  build  a  stronger  “BUSINESS  CASED  FOR  SYSTEMS  ENGINEERING*  (BCSE). 
Our  intent  is  to  provide  quantitative  iiformation  on  the  impact  of  SE  activities  on  project  performance.  This  information  will  aid 
organizations  arxi  individuals  utilizing  SE  to  focus  their  efforts  on  activities  that  produce  measureable  bertefits.  We  would  like 
your  organization  to  participate  in  this  study  by  completing  a  CONFIDENTIAL  and  ANONYMOUS  survey. 

In  2006.  the  SE  Effectiveness  Committee  of  the  NDIA  Systems  Engineering  Division  oor>ducted  the  Systems  Errgineering 
Effectiveness  Study  (SEES).  Using  survey  techniques,  ^is  study  identified  statistical  relationships  behveen  the  application  of 
specific  SE  practices  to  development  projects  and  the  perfbrmarice  of  those  projects,  as  measured  by  satisfaction  of  budget 
schedule,  and  requirements.  The  results  clearly  demonstrated  the  benefits  of  SE.  showing  that  projects  applying  the  least  SE 
performed  measurably  worse  than  projects  applying  the  most  SE.  The  study  also  identified  relationships  between  spedfic  SE 
practioes  (e.g..  requirements  development,  trade  study  performance,  architecture  development)  and  project  performance.  For 
riKxe  information  about  the  SEES,  you  may  download  several  reports,  papers,  and  presentations  from  the  BCSE  web  site  at 
vrvirw.sei.crrKi.edu/QO/bcse/. 

Based  on  the  success  of  the  prior  study,  in  2010  the  NDIA  embarked  on  the  BCSE  project  to  update  and  enhance  H  by  gathering 
data  from  a  larger  arxl  more  diverse  population.  The  BCSE  Project  Team  is  comprised  of  members  of  defense,  industry,  and 
academia  working  through  the  NDIA.  the  lEEE-AESS.  and  the  SEI.  This  project  has  been  coordinated  with  the  Office  of  the 
Secretary  of  Defense  through  the  Deputy  Assistant  Secretary  of  Defense.  Systems  Engineering  (DASD(SE)).  This  study  will 
survey  individual  product-producing  projects  to  assess  the  1 )  characteristics  of  the  project  2)  the  SE  activities  applied  to  the 
project  and  3)  the  resulting  project  performance.  Completion  of  the  survey  will  reqiare  approximately  30  minutes  for  each 
project  Like  the  prior  survey,  data  sectxity  and  corrfidentiality  wil  be  paramouit  ALL  DATA  WILL  BE  COLLECTED 
ANONYMOUSLY.  No  information  identifying  the  project,  organization,  or  respondent  will  be  requested.  The  Software 
Engineering  Institute,  a  DoD-sponsored  FFRDC.  will  do  the  data  colleclion  and  analysis.  Only  they  will  see  your  responses,  and 
only  statistical  summaries  of  the  aggregated  data,  untraceable  to  any  project  organization,  or  person,  will  be  released. 

Simiar  to  the  original  study,  those  who  participate  will  be  rewarded  with  early  access  to  more  detailed  levels  of  aggregated  data 
and  analysis  for  one  year  prior  to  these  results  being  published  for  everyone.  This  will  allow  you  to  assess  your 
organizatiorVproject  results  relative  to  the  rest  on  the  industry,  showing  your  strertgths  arfo  areas  of  weaknesses  that  shotid  be 
addressed. 

OUR  REQUEST  TO  YOU: 

By  this  Inquiry,  we  wish  to  reach  out  to  relevant  organizations  to  invite  them  to  participate  in  this  study.  If  your  organization 
executes  projects  that  produce  delivered  products,  we  ask  you  to  participate  in  this  study.  By  providing  the  data  requested 
below,  you  will  enable  us  to  contact  the  appropriate  decision  makers  within  your  organization  to  1 )  explain  this  important  study. 
2)  discuss  the  value  of  this  study  to  yotx  organization,  and  3)  request  the  participation  of  your  organization  in  the  study. 
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Please  take  just  a  minute  to  answer  the  following  six  questions  that  wtl  assist  us  in  reaching  out  to  potential  respondents  for  the 
survey.  Bas^  upon  the  information  that  you  pnwide.  we  twll  contact  the  designated  person  to  discuss  this  study.  If  they 
authorize  your  organization  to  participate,  an  invitation  will  be  sent  to  your  organization,  ertabiing  the  submission  of 
ANONYMOUS  AND  CONFIDENTIAL  responses  to  the  survey. 

If  you  have  any  questions,  piease  feel  to  contact  the  undersigned. 

Thank  you. 

William  F.  Lyons 

IEEE  AESS  Board  of  Governors 

253-657-5964 

william  f.lvonsiaboeino  com 

Alan  R.  Brown 

NDIA  SE  Effectiveness  Committee  Chair 

314-234-2337 

alan.r  brown2@boeino.com 

Joseph  P.  Elm 

Software  Engineering  Institute 
412-268-0132 
ielm@sei.cmu  edu 

Robert  C.  Rassa 
NDIA  SE  Division  Chair 
310-985-4962 
RCRassa@ravtheon.com 


Note:  Type  only  in  tbe  areas  designated  for  data  entry'.  Your  reply  will  be  automatically  processed  so  it  is  important 
that  the  form  or  the  message  is  not  altered  in  any  ot^  way. 


NDIA  /  IEEE  Systems  Engineering  Effectiveness  Study  -  Participation 
inquiry 

Tvoe  my  h  he  anas  decMnaaed  for  (BQ  entY  YIM  replv  ts  aukxTBKan  prccesEed.  Ttierelbie.  I  k 
tiyxon  hat  he  kxm  or  he  message  K  not  aagcahaiyohereiay.  For  moiereonnallon  about  Wrig 
on  ns  hm.  see  he  MoiiWiq: 


Email_Address:  '-^member  S  email  address^'' 

Type  any  contaruaon  or  fUTheis  and  IMB(S  ip  ID  255  diaraoere. 


I  Meniber_Name;  '-member’s  name-" 

I  Type  any  contsiaon  or  lunben  and  leoeis  ip  ID  255  diaraoers. 

I 

What  is  the  name  of  your 

organization::  _ 
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Type  any  connn<aon  or  tuTtteis  and  lelleis  If)  to  2SS  oiasoas. 


SegmenVOivtsion  of  your  _ 

organizatian  (if  ' 

appfccabte)r 

Type  any  contaiAn  or  lUTtiets  aid  lelfen  If)  to  2S5  dmcias. 

Name  (of  the  individual  _ 

mho  should  be 
approached  )= 

Type  aiy  laxmunn  or  luTtteis  aid  letfefs  If)  k>  2S5  Omtov 


Tide  (of  the  individual  _ 

mrtio  should  be 
approached):: 

Type  aiy  comaunn  or  njTOeis  aid  letfeis  If)  to  2S5  Oiaradav 

I 

Telephone  (of  the _ 

individual  who  shoUd  be 
approached):: 

Type  aiy  contaupon  or  nuTOen  aid  letfeis  If)  to  2S  diaaOeiv 


Email  (of  the  individual  _ 

a^o  should  be 
approached):: 

Type  ary  coimnann  or  lUTPen  aid  ietleis  If)  to  2SS  oiaracKre. 


Done?  Click  Send  to  submit  your  information. 
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Invitation  email 


Business  Case  for  Systems  Engineering 

From:  Business  Cdse  for  Systems  Engineering 
Senb  Tuesddy>  Noven^  22,  2011  11:49  AM 
To:  <invjtee's  emal  dddress> 

Subject:  SE  Effectiveness  Study 


Tr«^inNni«ii  i  Mr«»  'mci  atyxii'c-ujc'i 


Softwsre  Engineering  Instifirte  I  MnH-jni' MHIam 


FROM: 

The  Systems  Engineering  Effectiveness  Committee 
TO: 

<invitee  name> 

<invitee  organization> 

Welcome  to  the  Systems  Engineering  Effectiveness  Study,  a  component  of  the  “Business  Case  for  Systems 
Engineering”  (BCSE)  project  Through  inquiries  to  the  memberships  of  the  National  Defense  Industrial  Assodation 
(NDIA),  the  Institute  of  Electrical  and  Efectmnic  Engineers  Aerospace  and  Efectronic  Systems  Society  (lEEE/AESS), 
and  the  Intematiortal  Council  on  Systems  Engineering  (INCOSE),  you  have  been  nominated  as  a  potenbal 
respoTKlent  to  the  SE  Effectiver>ess  Survey. 

The  BCSE  Fhpject  Team  is  comprised  of  members  of  defense,  irxlustry,  arid  academia  working  through  the  NDIA, 
the  IEEE/AESS,  and  the  Software  Errgineermg  Institute  (SEI)  of  Carnegie  Mellon  University.  INCOSE  is  also 
promoting  this  project  to  its  members.  This  project  has  been  coordinated  with  the  Office  of  the  Secretary  of  Defense 
through  the  Deputy  Assistant  Secretary  of  Defense,  Systems  Engineering  (DASD(SE)). 

This  study  is  an  extension  of  the  2006  NDIA  Systems  Engineering  Division  conducted  the  Systems  Engineering 
Effectiveness  Study  (SEES),  which  was  successful  at  identifying  statistical  relationshiprs  between  the  appication  of 
specific  SE  practices  to  development  projects  and  the  performance  of  those  projects  through  anonymous  and 
survey  techniques.  The  results  of  this  prior  study  clearty  demonstrated  the  benefits  of  SE,  showing 

that; 

•  in  the  set  of  projects  applying  the  least  SE,  only  1 5%  delivered  the  highest  levels  of  performance 

•  in  the  set  of  projects  applying  the  most  SE,  56%  delivered  the  highest  levels  of  performance. 
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The  study  also  identified  relationships  between  specific  SE  practices  (e.g.,  requirements  developrrtent  and 
management,  trade  study  performance,  architecture  development)  and  protect  performance.  For  more  information 
atxxit  the  SEES,  you  may  download  several  reports,  papers,  and  presentations  from  the  BCSE  web  site  at 
WWW. sei. cmu.edu/QO/bcse/. 

We  are  asking  for  your  prarticipation  in  exteirsion  and  expansion  of  the  previous  study 

This  new  study  will  survey  individual  protects  within  organizations  like  yours  to  assess 

•  key  characteristics  of  the  protect, 

•  the  SE  activities  applied  to  the  protect,  and 

•  the  resulting  project  performance. 

Like  the  previous  study,  data  security  and  confidentiatitv  are  paramount  All  data  vmI  be  collected  anonymously.  No 
information  identifying  the  project,  organization,  or  respondent  wil  be  requested  or  collected.  The  Software 
Engineerirtg  Institute,  a  DoD-sponsored  FFRDC,  wll  do  the  data  colection  and  analysis.  Only  they  will  see  your 
responses,  and  only  aggregated  data,  untraceable  to  any  project,  organization,  or  person,  wll  be  released.  Simlar 
to  the  origirtal  study,  those  who  partcipate  will  be  rewarded  with  early  access  to  more  detailed  levels  of  aggregated 
data  and  analysis  for  one  year  prior  to  these  results  being  published  for  everyone.  This  will  allow  you  to  assess  your 
organization/project  results  relative  to  the  rest  on  the  indu!^,  showing  your  strengths  and  areas  of  weaknesses  that 
should  be  addressed. 

We  ask  that  you  randomly  select  one  or  more  product-producing  projects  within  your  organization  arxl  complete  a 
survey  for  each  of  them.  Please  do  not  attempt  to  select  the  only  the  best  (or  only  the  worst  performing  projects;  we 
seek  a  randomized  sample.  Projects  need  not  be  totaly  completed,  but  should  be  substantially  (>50%) 
complete.  Completion  of  the  survey  wil  require  approximately  30  minutes  for  each  project.  In  preparation  for 
completing  the  survey,  it  wil  be  helpful  to  assemble  the  following  data  regarding  your  project 

•  Initial  and  current  contract  value 

•  Initial  and  current  project  budget 

•  Initial  and  current  project  schedule 

•  Earned  value  data  (SPI  and  CPI) 

Once  you  have  assembled  this  data,  please  go  to 


httDs://feedback.sel.cmu.edu/2011  SE  Eff 

ectivenes^ 


to  access  a  copy  of  the  survey  questionnaire.  At  this  site,  you  will  receive  a  randomized  account  name  that  will 
enable  you  to  submit  an  anonymous  response. 
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Thank  you  for  your  participation  in  this  study.  Your  efforts  will  contribute  to  the  creation  of  a  business  case  that  will 
encourage  the  application  of  sound  SE  practices  to  produce  better  products  more  efiicientty  and  effectively. 

If  you  have  any  questions,  please  feel  to  contact  the  undersigned. 

Thank  you. 

William  F.  Lyons 
Alan  R.  Brown 
Joseph  P.  Bm 
Robert  C.  Rassa 


IEEE  AESS  Board  of  Governors  253-657-5984 

NDIA  SE  Effectiveness  Committee  Chair  314-234-2337 
Software  Engineering  Institute  412-268-9132 

NDIA  SE  Division  Chair  310-985-4962 


william  f  Ivonsiaboeino  com 

alan.r.brown2@boeinq.com 

ietm@8ei.cmu.edu 

RCRassa@ravtheon.com 
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Appendix  D  Details  of  the  Analysis  Process 


Project  Performance  (Perl) 

This  section  provides  a  summary  of  the  process  used  to  develop  Perf,  PerfC,  PerfD,  and  PerfS.  A 
detailed  discussion  of  this  process  is  provided  in  The  Business  Case  for  Systems  Engineering 
Study:  Assessing  Project  Performance  from  Sparse  Data  [Elm  2012]. 

Project  performance  (Perf)  is  assessed  using  the  survey  questions  shown  in  Table  12. 


Table  12:  Project  Performance  (Perf)  Assessment 


Q# 

Question 

Response  range 

Bl. 

What  is  the  current  total  contract  value  of  this  project? 

U.S.$ 

B2. 

What  was  the  initial  contract  value  of  this  project? 

u.s.$ 

B3. 

The  change  in  contract  value  is  primarily  due  to 

1=N/A;  No  change 

2=Change  in  tech,  scope 

3=Unplanned  increases 

4=Other 

B7. 

What  was  the  initial  total  budget  for  this  project? 

U.S.$ 

B8. 

What  is  the  current  total  budget  for  this  project? 

u.s.$ 

N7 

What  is  the  projected  cost  variance  at  completion  for  the 
current  contract  baseline? 

u.s.$ 

B9. 

The  change  in  budget  is  primarily  due  to 

1=N/A;  No  change 

2=Change  in  tech,  scope 

3=Unplanned  increases 

4=Customer  driven  increases 

5=Other 

N6 

When  is  the  EVMS  baseline  updated? 

•  Only  at  contract  initiation 

•  Whenever  a  contract  change  order  or 
renewal  is  received 

•  Incrementally  in  rolling  wave  planning 

•  Whenever  the  project  is  reprogrammed 
due  to  a  pre-determined  cost  or 
schedule  variance 

•  At  periodic  intervals 

•  Other 

N9 

What  is  the  current  cumulative  (or  final)  EVMS  CPI  for 
this  project? 

n 

09 

I  believe  that  my  customer  is  satisfied  with  this  project's 
performance  with  respect  to  cost. 

l=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

B4. 

What  is  the  current  total  planned  duration  of  this  project 
or  contract? 

Months 

B5. 

What  was  the  initial  total  planned  duration  of  this 
project  or  contract? 

Months 
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Q# 

Question 

Response  range 

B6. 

The  change  in  schedule  is  primarily  due  to 

1=N/A;  No  change 

2=Change  in  tech,  scope 

3=Unplanned  increases 

4=Cust.  driven  increases 

5=0ther 

N8 

What  is  the  projected  schedule  variance  at  completion 
for  the  current  contract  baseline? 

Months 

NIO 

What  is  the  current  cumulative  (or  final)  EVMS  SPI  for 
this  project? 

n 

04 

Overall,  this  project  is  performing  per  the  schedule 
established  in  the  current  IMS  approved  by  the  acquirer. 

l=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

05 

The  schedule  of  this  project’s  critical  path,  when 
compared  to  the  current  IMS  approved  by  the  acquirer  is 

>  6  months  late 

3-6  months  late 

1-3  months  late 

Within  +/-  1  month 

1-3  months  early 

3-6  months  early 

08 

I  believe  that  my  customer  is  satisfied  with  this  project's 
performance  with  respect  to  the  schedule. 

l=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

03 

Requirements  are  being  satisfied  and  remain  on  track  to 
be  satisfied  in  the  product  releases  as  originally  planned; 
they  are  not  being  deleted  or  deferred  to  later  releases. 

l=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

OlO 

I  believe  that  my  customer  is  satisfied  with  this  project's 
performance  with  respect  to  satisfaction  of  requirements. 

l=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

Project  performance  {Perf)  can  be  measured  and  decomposed  into 

Cost  performance  (PerfC  ) 

Schedule  performance  {PerfS  ) 

Technical  performance  (Pej/T  ) 

Each  of  these  factors  is  calculated  independently.  The  three  factors  are  then  combined  into  a 
weighted  summed  index  to  create  a  measure  of  project  performance  {Perf)  scaled  from  1  (very 
poor  performance)  to  5  (very  good  performance).*^ 


A  description  of  the  calculation  of  weighted  summed  indices  can  be  found  in  Section  4.1  on  page  10. 
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Project  Cost  Performance  (PerfC) 

PerfC  is  assessed  based  on  evaluations  of  three  factors: 

1.  estimated  cost  at  completion  (ECAC)  vs.  project  budget 

2.  EVMS  CPI 

3.  perception  of  customer  satisfaction  with  the  cost  performance  of  the  project 

Executing  a  project  within  its  budget  is  generally  considered  to  be  an  element  of  a  successful  pro¬ 
ject.  Thus,  comparing  the  ECAC  to  the  project  budget  is  a  reasonable  measure  of  project  perfor¬ 
mance.  However,  the  following  modifying  factors  must  be  considered. 

In  some  organizations,  the  budget  may  be  changed  to  reflect  significant  cost  ovenuns  or  project 
replanning.  In  these  cases,  comparison  of  the  ECAC  with  budget  is  no  longer  a  valid  measure  of 
project  performance.  To  remedy  this  defect,  we  derived  the  project  performance  assessment  as 

ECAC 

Cost  ratio  = - ; - ; — ; - 

initial  project  budget 

Sometimes  the  customer  issues  a  contract  change  order  that  modifies  the  project  scope,  contract 
value  (CV),  and/or  project  schedule.  In  response  to  such  a  change,  most  organizations  will  amend 
the  budget  to  reflect  the  change  in  the  expected  project  cost.  In  these  cases,  comparison  of  the 
ECAC  and  initial  budget  is  no  longer  a  valid  measure  of  project  performance.  Comparison  of  the 
ECAC  with  the  revised  budget  could  be  a  valid  measure,  unless  the  budget  revision  reflects  in¬ 
curred  or  anticipated  cost  ovenuns  as  previously  noted.  To  tease  apart  these  issues,  we  derive  the 
project  performance  assessment  from 

ECAC /initial  budget 

Cost  ratio  = - — — ; - — — 

revised  CV/ initial  CV 

The  cost  ratio  is  then  assessed  as  shown  in  Table  13. 


Table  13:  Budget-Based  Cost  Performance  Assessment 


Cost  Performance  Assessment 
(Cost_Perf_2) 

Criteria 

1 

(Cost  Ratio)  >  1.1 

2 

1.1  >  (Cost  Ratio)  >  1.05 

3 

1.05  >  (Cost  Ratio)  >  0.95 

4 

0.95  >  (Cost  Ratio)  >  0.90 

5 

0.90  >  (Cost  Ratio) 

An  assessment  of  EVMS  data  is  also  incorporated  into  the  measure  of  the  project’s  cost  perfor¬ 
mance.  CPI  appears  to  be  an  ideal  measure  of  project  performance,  but  again,  there  are  some 
modifying  factors  to  consider. 

EVMS  is  calculated  from  variances  from  a  baseline;  therefore,  it  is  highly  sensitive  to  revisions  in 
those  baselines.  The  questionnaire  asks  when  the  EVMS  baseline  is  updated.  The  response,  as 
shown  in  Table  14,  defines  how  CPI  data  are  used. 
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Table  14:  CPI  Data  Disposition 


The  EVMS  Baseline  is  updated  ... 

Other  conditions 

Disposition 

Only  at  contract  initiation 

Project  scope  has  not  changed 

Use  CPI.SPI 

Project  scope  has  changed 

Discard  CPI,  SPI 

Whenever  a  contract  change  order 
or  renewal  is  received 

Change  order  reflects  change  in 
scope 

Use  CPI.SPI 

Change  order  reflects  only 
recognition  of  cost  overrun 

Discard  CPI,  SPI 

Incrementally  in  rolling  wave 
planning 

Discard  CPI,  SPI 

Whenever  the  project  is 
reprogrammed  due  to  a  pre¬ 
determined  cost  or  schedule 

Discard  CPI,  SPI 

variance 

At  periodic  intervals 

Discard  CPI,  SPI 

Other 

Per  analyst  evaluation 

If  CPI  data  are  used,  they  are  assessed  as  shown  in  Table  15. 


Table  15:  CPI-Based  Cost  Performance 


Cost  Performance  Assessment 
(Cost_Perf_1) 

Criteria 

1 

0.90  >  {CPT) 

2 

0.95  >  (CPI)  >  0.90 

3 

1.05  >  (CPI)  >  0.95 

4 

1.1  >  (CPI)  >  1.05 

5 

(CPI)  >  1.1 

The  final  factor  evaluated  for  assessment  of  project  cost  performance  is  the  perception  of  the  cus¬ 
tomer’s  satisfaction  with  the  cost  performance  of  the  project,  assessed  as  shown  in  Table  16. 


Table  16:  Customer-Satisfaction-Based  Cost  Performance  Assessment 


Cost  Performance  Assessment 
(Cost_Perf_3) 

1  believe  that  my  customer  is  satisfied  with  this  project's 
performance  with  respect  to  cost 

1.00 

Strongly  disagree 

2.33 

Disagree 

3.67 

Agree 

5.00 

Strongly  Agree 

These  three  assessment  factors  are  combined  into  a  weighted  summed  index,  PerfC,  to  provide  an 
assessment  of  project  cost  performance  scaled  from  1  (very  poor  performance)  to  5  (very  good 
performance). 
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Project  Schedule  Performance  (Pe/fS) 

PerfS  is  assessed  based  on  evaluations  of  four  factors: 

1 .  the  difference  between  the  initial  completion  date  and  the  current  estimated  completion  date 

2.  the  Schedule  Performance  Index  (SPI)  from  an  Earned  Value  Management  System  (EVMS) 

3.  variance  from  the  approved  integrated  master  schedule  (IMS) 

4.  the  respondent’s  impression  of  the  degree  of  customer  satisfaction  with  the  project  cost 

In  comparing  the  initial  and  current  estimated  completion  dates,  we  encountered  the  same  chal¬ 
lenge  seen  with  the  cost  data  discussed  in  the  Project  Cost  Performance  (PerfC)  section  on  page 
133 — changes  in  the  scope  of  the  project.  For  cases  where  the  scope  of  the  project  has  remained 
stable,  we  calculated  a  schedule  performance  assessment  as 

Sched_current  +  ECDjvariance 

Schedule  ratio  = - ; — ; - - - 

Schedjnitial 


For  projects  with  a  scope  modified  by  a  contract  change  order,  we  used  the  change  in  contract 
value  (CV)  as  a  correction  factor  for  our  schedule  performance  assessment: 


Schedule  ratio  = 


Scched_current  +  ECDjvariance 
Schedjnitial 


CV  _current 
CVJnitial 


The  schedule  ratio  is  then  assessed  as  shown  in  Table  17. 


Table  17:  Project-Duration-Based  Schedule  Performance  Assessment 


Schedule  Performance  Assessment 
(Sched_Perf_2a) 

Criteria 

1 

(Schedule  Ratio)  >  1.1 

2 

1.1  >  (Schedule  Ratio)  >  1.05 

3 

1.05  >  (Schedule  Ratio)  >  0.95 

4 

0.95  >  (Schedule  Ratio)  >  0.90 

5 

0.90  >  (Schedule  Ratio) 

Our  second  assessment  of  schedule  performance  is  derived  from  EVMS  data  using  the  SPI. 
Again,  as  discussed  in  the  Project  Cost  Performance  (PerfC)  section  on  page  133,  the  validity  of 
the  SPI  data  is  affected  by  the  criteria  used  to  manage  the  EVMS  baselines.  Table  14  illustrates 
the  validity  and  the  disposition  of  SPI  data.  If  the  SPI  data  are  acceptable,  they  are  assessed  as 
shown  in  Table  18. 


Table  18:  SPI-Based  Schedule  Performance 


Schedule  Performance  Assessment 
{Sched_Perf_1) 

Criteria 

1 

0.90  >  (SPI) 

2 

0.95  >  (SPI)  >  0.90 

3 

1.05  >  (SPI)  >  0.95 

4 

1.1  >  (SPI)  >  1.05 

5 

(SPI)  >  1.1 
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Our  third  assessment  of  schedule  performance  is  based  on  deviation  from  the  customer-approved 
IMS,  based  on  the  question  in  Table  19. 


Table  19:  IMS-Based  Schedule  Performance  Assessment 


Q# 

Question 

Response 

Assessment 

05. 

The  schedule  of  this  project’s  critical  path, 

Degree  of 

1  =  >  6  months  late 

when  compared  to  the  current  IMS  ap¬ 
proved  by  the  acquirer  is  ... 

deviation 

2=  3  -  6  months  late 

3=  1-  3  months  late 

4=  Within  +/- 1  month 

5=  1  -  3  months  early 

6=  3  -  6  months  early 

The  final  factor  evaluated  for  the  assessment  of  project  schedule  performance  is  the  perception  of 
the  customer’s  satisfaction  with  the  schedule  performance  of  the  project,  assessed  as  shown  in 
Table  20. 

Table  20:  Customer-Satisfaction-Based  Schedule  Performance  Assessment 


Schedule  Performance  Assessment 
{Sched_Perf_4) 

I  believe  that  my  customer  is  satisfied  with  this  project's 
performance  with  respect  to  schedule 

1.00 

Strongly  disagree 

2.33 

Disagree 

3.67 

Agree 

5.00 

Strongly  Agree 

These  four  assessment  factors  are  combined  into  a  weighted  summed  index,  PerfS,  to  provide  an 
assessment  of  project  schedule  performance  scaled  from  1  (very  poor  performance)  to  5  (very 
good  performance). 

Project  Technical  Performance  (PerfT) 

PerfT  is  assessed  based  on  evaluations  of  two  factors: 

1 .  satisfaction  of  requirements 

2.  perception  of  customer  satisfaction  with  the  technical  performance  of  the  project 


PerfT  is  assessed  based  on  the  questions  in  Table  21. 
Table  21:  PerfT  Assessment 


Q# 

Question 

Response 

Assessment 

03 

Requirements  are  being  satisfied  and  re¬ 
main  on  track  to  be  satisfied  in  the  product 
releases  as  originally  planned;  they  are  not 
being  deleted  or  deferred  to  later  releases. 

Degree  of 
agreement 

1  =Strongly  disagree 

2.33=Disagree 

3.67=Agree 

5=Strongly  Agree 

O10 

1  believe  that  my  customer  is  satisfied  with 
this  project's  performance  with  respect  to 
satisfaction  of  requirements. 

Degree  of 
agreement 

1  =Strongly  disagree 

2.33=Disagree 

3.67=Agree 

5=Strongly  Agree 

These  responses  are  combined  into  a  weighted  summed  index,  PerfT,  to  provide  an  assessment  of 
project  technical  performance  scaled  from  1  (very  poor  performance)  to  5  (very  good  perfor¬ 
mance). 
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PROJECT  CHALLENGE  (PC) 


PC  information  was  collected  through  responses  to  the  questions  shown  in  Table  22.  The  response 
to  each  question  was  assessed  as  shown  in  the  Assessment  column. 


Table  22:  PC  Assessment 


Q# 

Question 

Response 

Assessment 

A1. 

The  project  is  challenging  because  there  is 
no  precedent  for  what  is  being  done. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

A2. 

This  project  is  challenging  because 
significant  constraints  are  placed  on  the 
quality  attributes  of  the  product. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

A3. 

The  project  is  challenging  because  the  size 
of  the  development  effort  is  large. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

A4. 

The  project  is  challenging  because  the 
technology  needed  for  this  project  is  not 
mature  or  otherwise  poses  a  high  risk. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

A5. 

The  project  is  challenging  because  there 
are  extensive  needs  for  interoperability  with 
other  systems. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

A6. 

The  project  is  challenging  because  there 
are  insufficient  resources  available  to 
support  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

A7. 

The  project  is  challenging  because  there 
are  insufficient  skills  and  subject  matter 
expertise  available  to  support  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

A9. 

In  the  past,  this  project  team  has  NOT 
successfully  completed  projects  of  similar 
scope. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

A10. 

The  requirements  supplied  by  the  customer 
for  this  project  are  NOT  well-defined. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

All. 

The  requirements  supplied  by  the  customer 
for  this  project  have  NOT  changed 
sufficiently  to  generate  a  significant  impact 
on  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

Cl 

This  organization  has  NOT  successfully 
completed  projects  similar  in  scope  to  this 
one  in  the  past. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D20 

The  acquirer  HAS  NOT  provided  this 
project  with  a  SE  Plan  in  a  timely  manner. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment 

A12. 

What  percentage  of  the  customer  technical 
requirements  were  marked  “TBD”  or  equiv. 
at  time  of  contract  award? 

0-100% 

1  «-  5%  >  (undef  reqts) 

2  «-  10%  >  {undef  reqts)  >  5% 

3  «-  20%  >  {undef  reqts)  >  10% 

4  «-  {undef  reqts)  >  20% 

A13. 

What  percentage  of  the  customer’s 
technical  requirements  are  currently 
marked  “TBD”  or  equiv? 

0-100% 

1  «-  5%  >  {undef  reqts) 

2  «-  10%  >  {undef  reqts)  >  5% 

3  «-  20%  >  {undef  reqts)  >  10% 

4  «-  {undef  reqts)  >  20% 

B1. 

What  is  the  current  total  contract  value  of 
this  project? 

u.s.$ 

1  <-  $le6  >  {CV) 

2  <-  $le7  >  {CV)  >  $le6 

3  <-  $le8  >  {CV)  >  $le7 

4  <-  {CV)  >  $le8 

B2. 

What  was  the  initial  contract  value  of  this 
project? 

u.s.$ 

1  <-  $le6  >  {CV) 

2  <-  $le7  >  {CV)  >  $le6 

3  <-  $le8  >  {CV)  >  $le7 

4  <-  {CV)  >  $le8 

B4. 

What  is  the  current  total  planned  duration 
of  this  project  or  contract? 

Months 

1  «-  30  >  {Duration) 

2  «-  60  >  {Duration)  >  30 

3  «-  120  >  {Duration)  >  60 

4  «-  {Duration)  >  120 

B5. 

What  was  the  initial  total  planned  duration 
of  this  project  or  contract? 

Months 

1  ->  30  >  {Duration) 

2  ->  60  >  {Duration)  >  30 

3  ->  120  >  {Duration)  >  60 

4  -»  {Duration)  >  120 

B7. 

What  was  the  initial  total  budget  for  this 
project? 

U.S.$ 

1  «-  $le6  >  {budget) 

2  <r-  $le7  >  {budget)  >  $le6 

3  «-  $le8  >  {budget)  >  $le7 

4  «-  {budget)  >  $le8 

B8. 

What  is  the  current  total  budget  for  this 
project? 

U.S.$ 

1  «-  $le6  >  {budget) 

2  <r-  $le7  >  {budget)  >  $le6 

3  «-  $le8  >  {budget)  >  $le7 

4  «-  {budget)  >  $le8 

BIO. 

How  many  contract  change  orders  have 
been  received? 

N 

All  assessments  were  scaled  from  1  to  4.  The  results  of  all  assessments  were  then  combined  into  a 
weighted  summed  index,  PC,  to  create  an  overall  assessment  of  project  challenge  scaled  from  1 
(very  low  challenge)  to  4  (very  high  challenge). 

SYSTEMS  ENGINEERING  CAPABILITY  (SEC) 

SEC  is  a  measure  of  the  SE  activities  applied  to  each  project.  In  addition  to  assessing  the  total  SE 
activities  {SEC-Total}  applied,  we  also  are  able  to  assess  the  SE  activities  applied  in  each  of  the 
SE  process  groups  shown  in  Table  6.  For  each  of  these  process  groups,  this  section  identifies  the 
questions  and  criteria  that  produce  the  assessment. 
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Integrated  Product  Team  Capability  (SEC-IPT) 

The  project’s  use  of  integrated  product  teams  was  assessed  through  the  questions  shown  in  Table 
23. 


Table  23:  SEC-IPT  Assessment 


Q# 

Question 

Response 

Assessment  criteria 

El 

This  project  makes  effective  use  of  IPTs. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

E2 

My  acquirer  participates  in  my  iPTs  for  this 
project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

E3 

My  suppliers  actively  participate  in  my  IPTs. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

Q# 

Question 

Response 

Assessment  criteria 

E4 

This  project  has  an  IPT  with  assigned 
responsibility  for  SE. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

E5 

This  project  has  SE  representation  on  each 
IPT. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-IPT  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Project  Planning  Capability  (SEC-PP) 

The  project’s  application  of  best  practices  in  Project  Planning  was  assessed  through  the  questions 
shown  in  Table  24. 


Table  24:  SEC-PP  Assessment 


Q# 

Question 

Response 

Assessment  criteria 

A14 

Do  you  separately  budget  and  track 

Systems  Engineering  activities? 

Yes,  No,  Don't 
Know 

4= Yes 

1=No 

1=Don’t  Know 

D1 

This  project  utilizes/utilized  a  documented 
set  of  Systems  Engineering  processes  for 
the  planning  and  execution  of  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D2 

This  project  has/had  an  accurate  and  up-to- 
date  WBS  that  included  task  descriptions 
and  work  package  descriptions. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D3 

This  project  has/had  an  accurate  and  up-to- 
date  WBS  that  was  based  on  the  product 
structure. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

D4 

This  project  has/had  an  accurate  and  up-to- 
date  WBS  that  was  developed  with  the 
active  participation  of  those  who  perform 
the  systems  engineering  activities. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D5 

This  project  has/had  an  accurate  and  up-to- 
date  WBS  that  was  developed  and 
maintained  with  the  active  participation  of 
all  relevant  stakeholders. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D6 

This  project’s  Technical  Approach  is 
complete,  accurate  and  up-to-date. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D7 

This  project’s  Technical  Approach  is 
developed  and  maintained  with  the  active 
participation  of  those  who  perform  the 
Systems  Engineering  activities. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D8 

This  project’s  Technical  Approach  is 
developed  and  maintained  with  the  active 
participation  of  all  appropriate  functional 
stakeholders. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D9 

This  project  has  a  top-level  plan,  such  as 
an  IMP,  that  is  an  event-driven  plan  (i.e., 
each  accomplishment  is  tied  to  a  key 
project  event). 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

DIO 

This  project  has  a  top-level  plan,  such  as 
an  IMP,  that  documents  significant 
accomplishments  with  pass/fail 
accomplishment  criteria  for  both  business 
and  technical  elements  of  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

Dll 

This  project  has  a  top-level  plan,  such  as 
an  IMP,  that  is  consistent  with  the  WBS. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D12 

This  project  has  an  integrated  event-based 
schedule  that  is  structured  as  a  networked, 
multi-layered  schedule  of  project  tasks 
required  to  complete  the  work  effort. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D13 

This  project  has  an  integrated  event-based 
schedule  that  contains  a  compilation  of  key 
technical  accomplishments  (e.g.,  a  SE 

Master  Schedule). 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D14 

This  project  has  an  integrated  event-based 
schedule  that  references  measurable 
criteria  (usually  contained  in  the  IMP) 
required  for  successful  completion  of  key 
technical  accomplishments. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D15 

This  project  has  an  integrated  event-based 
schedule  that  is  consistent  with  the  WBS. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D16 

This  project  has  an  integrated  event-based 
schedule  that  identifies  the  critical  path  of 
the  program  schedule. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

D17 

This  project  has  a  plan  or  plans  for  the 
performance  of  technical  reviews  with 
defined  entry  and  exit  criteria  throughout 
the  lifecycle  of  the  project. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D18 

The  SE  function  actively  participates  in  the 
development  and  updates  of  the  project 
planning. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D19 

Those  who  perform  SE  activities  actively 
participate  in  tracking/reporting  of  task 
progress. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

D20 

The  acquirer  provided  this  project  with  a  SE 
Plan  in  a  timely  manner. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D21 

This  project  has  a  plan  or  plans  that  include 
details  of  the  management  of  the  integrated 
technical  effort  across  the  project  (e.g.,  a 

SE  Mgt.  Plan  or  a  SE  Plan). 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

D22 

The  SEMP  developed  by  the  project  team 
is  aligned  and  consistent  with  the  SE  Plan 
provided  by  the  acquirer. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-PP  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Project  Monitoring  and  Controi  Capabiiity  (SEC-PMC) 

The  project’s  application  of  best  practices  in  Project  Monitoring  and  Control  was  assessed 
through  the  questions  shown  in  Table  25. 


Table  25:  SEC-PMC  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

K6 

This  project  has  a  documented  and 
practiced  review  process  that  tracks  action 
items  to  closure. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

N1 

This  project  creates  and  manages  cost  and 
schedule  baselines. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

N2 

EVMS  data  are  available  to  decision 
makers  in  a  timely  manner. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

N3 

The  requirement  to  track  and  report  EVMS 
data  is  levied  on  the  project’s  suppliers. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

N4 

Variance  thresholds  for  the  CPI  and  SPI  are 
defined,  documented,  and  used  to 
determine  when  corrective  action  is 
needed. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

N5 

The  EVMS  is  linked  to  the  technical  effort 
through  the  WBS,  and  the  IMS. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

N6 

When  is  the  EVMS  baseline  updated? 

3<-Only  at  contract  initiation 
4<-Whenever  a  contract  change  order 
or  renewal  is  received 

1<-lncrementally  in  rolling  wave 
planning 

1<-Whenever  the  project  is 
reprogrammed  due  to  a  pre-determined 
cost  or  schedule  variance 

1<-At  periodic  intervals 

Per  analyst  evaluation<-Other 

A14. 

Do  you  separately  budget  and  track 

Systems  Engineering  activities? 

Yes,  No,  Don't 

Know 

4<-Yes 

I^No 

1<-Don't  Know 

D19 

Those  who  perform  SE  activities  actively 
participate  in  tracking/reporting  of  task 
progress. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

06 

This  project  collects  and  tracks  (or  will 
collect  and  track)  reports  of  problems  from 
fielded  items. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

07 

This  project  conducts  (or  will  conduct) 
engineering  assessments  of  all  field  trouble 
reports. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-PMC  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Risk  Management  Capability  (SEC-RSKM) 

The  project’s  application  of  best  practices  in  Risk  Management  was  assessed  through  the  ques¬ 
tions  shown  in  Table  26. 


Table  26:  SEC-RSKM  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

FI 

This  project  has  a  Risk  Management 

Degree  of 

1=Strongly  disagree 

process  that  creates  and  maintains  an 
accurate  and  up-to-date  list  of  risks 
affecting  the  project. 

agreement 

2=Disagree 

3=Agree 

4=Strongly  Agree 

F2 

This  project  has  a  Risk  Management 

Degree  of 

1  =Strongly  disagree 

process  that  creates  and  maintains  up-to- 
date  documentation  of  risk  mitigation  plans 
and  contingency  plans  for  selected  risks. 

agreement 

2=Disagree 

3=Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

F3 

This  project  has  a  Risk  Management 
process  that  monitors  and  reports  the 
status  of  risk  mitigation  activities  and 
resources. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

F4 

This  project  has  a  Risk  Management 
process  that  assesses  risk  against 
achievement  of  an  event-based  schedule. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

F5 

This  project's  Risk  Management  process  is 
integrated  with  project  decision-making. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

F6 

This  project's  Risk  Management  process  is 
integrated  with  program  cost  and/or  earned 
value  management. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

F7 

This  project's  Risk  Management  process  is 
integrated  with  program  scheduling. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

F8 

This  project's  Risk  Management  process 
integrates  subcontract  or  supplier  risk 
management  processes. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-RSKM  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Requirements  Development  and  Management  Capability  (SEC-REQ) 

The  project’s  application  of  best  practices  in  Requirements  Development  and  Requirements  Man¬ 
agement  was  assessed  through  the  questions  shown  in  Table  27. 


Table  27:  SEC-REQ  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

G1 

This  project  maintains  an  up-to-date  and 
accurate  listing  of  all  requirements  specified 
by  the  customer,  to  include  regulatory, 
statutory,  and  certification  requirements. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G2 

This  project  maintains  an  up-to-date  and 
accurate  listing  of  all  requirements  derived 
from  those  specified  by  the  customer. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G3 

This  project  maintains  up-to-date  and 
accurate  documentation  clearly  reflecting 
the  hierarchical  allocation  of  both  customer 
and  derived  requirements  to  each  element 
(subsystem,  component,  etc.)  of  the  system 
in  the  configuration  baselines. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G4 

This  project  documents  and  maintains 
accurate  and  up-to-date  descriptions  of 
operational  concepts  and  their  associated 
scenarios. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

G5 

This  project  documents  and  maintains 
accurate  and  up-to-date  descriptions  of  use 
cases  (or  their  equivalent). 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G6 

This  project  documents  and  maintains 
accurate  and  up-to-date  descriptions  of 
product  installation,  maintenance  and 
support  concepts. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G7 

This  project  has  documented  criteria  for 
identifying  authorized  requirements 
providers  to  avoid  requirements  creep  and 
volatility. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G8 

This  project  has  documented  criteria  (e.g., 
cost  impact,  schedule  impact,  authorization 
of  source,  contract  scope,  requirement 
quality)  for  evaluation  and  acceptance  of 
requirements. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G9 

The  requirements  for  this  project  are 
approved  in  a  formal  and  documented 
manner  by  relevant  stakeholders. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G10 

This  project  performs  and  documents 
requirements  impact  assessments  for 
proposed  requirements  changes. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G11 

This  project  develops  and  documents 
project  requirements  based  on  stakeholder 
needs,  expectations,  and  constraints. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G12 

This  project  has  an  accurate  and  up-to-date 
requirements  management  system. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G13 

For  this  project,  the  requirements 
documents  are  managed  under  a 
configuration  control  process. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

G14 

For  this  project,  the  requirements 
documents  are  accessible  to  all  relevant 
project  staff. 

Degree  of 
agreement 

1=Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-REQ  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 
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Trade  Studies  Capability  (SEC-TRD) 


The  project’s  application  of  best  practices  in  Trade  Studies  was  assessed  through  the  questions 
shown  in  Table  28. 


Table  28:  SEC-TRD  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

HI 

Stakeholders  impacted  by  trade  studies  are 
involved  in  the  development  and 
performance  of  those  trade  studies. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

H2 

This  project  performs  and  documents  trade 
studies  between  alternate  solutions  in  a 
timely  manner,  and  based  on  definitive  and 
documented  selection  criteria. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

H3 

Documentation  of  trade  studies  is 
maintained  in  a  defined  repository  and  is 
accessible  to  all  relevant  project  staff. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-TRD  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Architecture  Capability  (SEC-ARCH) 

The  project’s  application  of  best  practices  in  Product  Architecture  was  assessed  through  the  ques¬ 
tions  shown  in  Table  29. 


Table  29:  SEC-ARCH  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

i1 

This  project  maintains  accurate  and  up-to- 
date  descriptions  (e.g.  interface  control 
documents,  models,  etc.)  defining 
interfaces  in  detail. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

i2 

Interface  definition  descriptions  are 
maintained  in  a  designated  location,  under 
configuration  management,  and  accessible 
to  all  who  need  them. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

i3 

For  this  project,  the  product  high-level 
structure  is  documented,  kept  up  to  date, 
and  managed  under  configuration  control. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

i4 

For  this  project,  the  product  high-level 
structure  is  documented  using  multiple 
views  (e.g.  functional  views,  module  views, 
etc.). 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

i5 

For  this  project,  the  product  high-level 
structure  is  accessible  to  all  relevant  project 
personnel. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-ARCH  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 
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Product  Integration  Capability  {SEC-PI) 


The  project’s  application  of  best  practices  in  Product  Integration  was  assessed  through  the  ques¬ 
tions  shown  in  Table  30. 


Table  30:  SEC-PI  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

J1 

This  project  has  accurate  and  up-to-date 

Degree  of 

1  =Strongly  disagree 

documents  defining  its  product  integration 

agreement 

2=Disagree 

process,  plans,  criteria,  etc.  throughout  the 

3= Agree 

lifecycle. 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-PI  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Verification  Capability  (SEC-VER) 

The  project’s  application  of  best  practices  in  Verification  was  assessed  through  the  questions 
shown  in  Table  31. 


Table  31:  SEC-VER  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

K1 

This  project  has  accurate  and  up-to-date 
documents  defining  the  procedures  used  for 
the  test  and  verification  of  systems  and 
system  elements. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

K2 

This  project  has  accurate  and  up-to-date 
documents  defining  acceptance  criteria  used 
for  the  verification  of  systems  and  system 
elements. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

K3 

This  project  has  a  documented  and  practiced 
review  process  for  work  products  that 
defines  entry  and  exit  criteria. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

K4 

This  project  has  a  documented  and  practiced 
review  process  that  includes  training  the 
reviewers  to  conduct  reviews. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

K5 

This  project  has  a  documented  and  practiced 
review  process  that  defines  criteria  for  the 
selection  of  work  products  for  review. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

K6 

This  project  has  a  documented  and  practiced 
review  process  that  tracks  action  items  to 
closure. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

K7 

This  project  has  a  documented  and  practiced 
review  process  that  addresses  identified 
risks  and  risk  mitigation  activities  during 
reviews. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

K8 

This  project  has  a  documented  and  practiced 
review  process  that  examines  completeness 
of  configuration  baselines. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 
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Q# 

Question 

Response 

Assessment  criteria 

K9 

This  project  conducts  non-advocate  reviews 

Degree  of 

1  =Strongly  disagree 

and  documents  results,  issues,  action  items, 

agreement 

2=Disagree 

risks,  and  risk  mitigations. 

3= Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-VER  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Validation  Capability  {SEC-VAL) 


The  project’s  application  of  best  practices  in  Validation  was  assessed  through  the  questions  shown 
in  Table  32. 

Table  32:  SEC-VAL  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

LI 

This  project  has  accurate  and  up-to-date 

Degree  of 

1  =Strongly  disagree 

documents  defining  the  procedures  used  for 
the  validation  of  systems  and  system 
elements. 

agreement 

2=Disagree 

3=Agree 

4=Strongly  Agree 

L2 

This  project  has  accurate  and  up-to-date 

Degree  of 

1  =Strongly  disagree 

documents  defining  acceptance  criteria  used 
for  the  validation  of  systems  and  system 
elements. 

agreement 

2=Disagree 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-VAL  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Configuration  Management  Capability  (SEC-CM) 

The  project’s  application  of  best  practices  in  Configuration  Management  was  assessed  through  the 
questions  shown  in  Table  33. 


Table  33:  SEC-CM  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

G13 

For  this  project,  the  requirements  documents 
are  managed  under  a  configuration  control 
process. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

Ml 

This  project  maintains  a  listing  of  items 
managed  under  configuration  control. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

M2 

This  project  has  a  configuration  management 
system  that  charters  a  CCB  to  disposition 
change  requests. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3= Agree 

4=Strongly  Agree 

M3 

This  project  maintains  records  of  requested 
and  implemented  changes  to  configuration- 
managed  items. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 

M4 

This  project  creates  and  manages 
configuration  baselines. 

Degree  of 
agreement 

1  =Strongly  disagree 

2=Disagree 

3=Agree 

4=Strongly  Agree 
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The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
SEC-CM  scaled  from  1  (very  low  capability)  to  4  (very  high  capability). 

Total  Systems  Engineering  Capability  (SEC-Total) 

The  capability  subcategories  of  D3.1  through  D3.1 1  were  combined  into  a  weighted  summed  in¬ 
dex  to  produce  a  measure  of  total  systems  engineering  capability  {SEC-Total)  scaled  from  1  (very 
low  capability)  to  4  (very  high  capability). 

OTHER  FACTORS 

Factors  other  than  project  challenge  may  also  influence  project  performance.  Factors  considered 
in  the  survey  included: 

•  Prior  Experience — ^the  degree  of  experience  that  the  organization  had  with  similar  pro¬ 
jects 

•  Contract  Type — fixed  price,  cost  reimbursable,  or  other  contract  type  governing  the  pro¬ 
ject 

•  SE  Organization — SE  organized  as  a  separate  department  or  distributed  throughout  the 
organization 

•  Percentage  Complete — ^the  percentage  of  the  project  that  has  been  complete 

•  SE  Content — ^the  percentage  of  the  non-recurring  engineering  effort  dedicated  to  systems 
engineering 

Prior  Experience  (EXP) 

The  project’s  prior  experience  with  similar  projects  was  assessed  through  the  questions  shown  in 


Table  34. 

Table  34:  EXP  Assessment  Questions 


Q# 

Question 

Response 

Assessment  criteria 

A9. 

In  the  past,  this  project  team  has 

Degree  of 

1  =Strongly  disagree 

successfully  completed  projects  of  similar 

agreement 

2=Disagree 

scope. 

3=Agree 

4=Strongly  Agree 

Cl 

This  organization  has  successfully 

Degree  of 

1  =Strongly  disagree 

completed  projects  similar  in  scope  to  this 

agreement 

2=Disagree 

one  in  the  past. 

3=Agree 

4=Strongly  Agree 

The  assessed  values  are  combined  into  a  weighted  summed  index  to  create  the  assessment  for 
EXP  scaled  from  1  (very  low  experience)  to  4  (very  high  experience). 
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Contract  Type 

The  type  of  contract  governing  the  project  was  assessed  through  the  question  shown  in  Table  35. 


Table  35:  Contract  Type  Question 


Q# 

Question 

Response 

B13. 

What  type  of  contract{s)  was  awarded  for 
this  project? 

This  is  a  fixed-price  contract  -  the  total  contract  value  is  primarily 
determined  by  the  initial  contract,  (e.g.,  FFP,  FPIF,  FFP-LOE). 

This  is  a  cost-reimbursable  contract  -  the  total  contract  value  is 
primarily  determined  by  my  cost  of  executing  the  contract  (e.g., 

CPFF,  CPAF,  CPIF). 

This  contract  does  not  fit  the  categories  listed  above. 

The  response  is  used  to  categorize  projects  and  segment  the  data  set  to  assess  the  impact  of  con¬ 
tract  type. 

SE  Organization 


The  structure  of  the  SE  operations  within  the  organization  was  assessed  through  the  question 
shown  in  Table  36. 

Table  36:  SE  Organization  Question 


Q# 

Question 

Response 

C2. 

Within  this  organization 

Systems  engineering  skills  and  responsibilities  are  contained  in  a 
separate  department. 

Systems  engineering  skills  and  responsibilities  are  distributed 
throughout  other  departments. 

The  response  is  used  to  categorize  projects  and  segment  the  data  set  to  assess  the  impact  of  the 
structure  of  SE  operations. 

Percentage  Compiete 

The  degree  of  completion  of  the  project  was  assessed  through  the  question  shown  in  Table  37. 


Table  37:  Percentage  Complete  Question 


Q# 

Question 

Response 

B12. 

What  is  the  current  completion  status  of  this 
project? 

0-100% 

The  response  is  used  to  categorize  projects  and  segment  the  data  set  to  assess  the  impact  of  per¬ 
centage  complete. 
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SE  Content 


The  magnitude  of  the  SE  effort,  as  a  percentage  of  the  total  project  non-recurring  engineering  ef¬ 
fort,  was  assessed  through  the  question  shown  in  Table  38. 


Table  38:  Percentage  SE  Question 


Q# 

Question 

Response 

B12. 

Approximately  what  percentage  of  non¬ 

0-100% 

recurring  engineering  (NRE)  does  Systems 

Engineering  represent? 

The  response  is  used  to  categorize  projects  and  segment  the  data  set  to  assess  the  impact  of  per¬ 
centage  SE. 
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Acronyms  and  Abbreviations 


BCSE 

CMMI 

COTS 

CPAF 

CPFF 

CPIF 

CPI 

CV 

DCAA 

DCMA 

DoD 

ECAC 

EVMS 

EXP 

FFP 

FFP-LOE 

FFRDC 

FPIF 

lEEE-AESS 

IEEE 

IMP 

IMS 

INCOSE 

n/a 

NASA 

NDIA 

NDIA-SED 

PC 

Perf 

PerfC 

PerfS 

PerfT 

RFP 

SE 

SEC 

SEC-ARCH 


business  case  for  systems  engineering 

Capability  Maturity  Model  Integration 

commercial  off-the-shelf 

cost  plus  award  fee 

cost  plus  fixed  fee 

cost  plus  incentive  fee 

EVMS  Cost  Performance  Index 

contract  value 

Defense  Contract  Audit  Agency 
Defense  Contract  Management  Agency 
(U.S.)  Department  of  Defense 
estimated  cost  at  completion 
Earned  Value  Management  System 
prior  experience 
firm  fixed  price 

firm  fixed  price — level  of  effort 

federally  funded  research  and  development  center 

fixed  price  incentive  fee 

Institute  of  Electrical  and  Electronic  Engineers  —  Aerospace  and  Electronic  Sys¬ 
tems  Society 

Institute  of  Electrical  and  Electronic  Engineers 
integrated  master  plan 
integrated  master  schedule 
International  Council  on  Systems  Engineering 
not  applicable 

National  Aeronautics  and  Space  Administration 
National  Defense  Industrial  Association 

National  Defense  Industrial  Association  Systems  Engineering  Division 

project  challenge 

project  performance 

project  cost  performance 

project  schedule  performance 

project  technical  performance 

request  for  proposal 

systems  engineering 

systems  engineering  capability 

product  architecture  systems  engineering  capability 
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SEC-CM 

SEC-IPT 

SEC-PI 

SEC-PMC 

SEC-PP 

SEC-REQ 

SEC-RSKM 

SEC-Total 

SEC-TRD 

SEC-VAL 

SEC-VER 

SEI 

SEMP 

SoS 

SPI 

WBS 


configuration  management  systems  engineering  capability 
integrated  project  team  systems  engineering  capability 
product  integration  systems  engineering  capability 
project  monitoring  and  control  systems  engineering  capability 
project  planning  systems  engineering  capability 

requirements  development  and  management  systems  engineering  capability 

risk  management  systems  engineering  capability 

total  SE  capability  applied  to  a  project 

trade  study  systems  engineering  capability 

validation  systems  engineering  capability 

verification  systems  engineering  capability 

Software  Engineering  Institute 

systems  engineering  master  plan 

system  of  systems 

EVMS  Schedule  Performance  Index 
work  breakdown  structure 
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